Systems and methods for concurrently playing multiple images from a storage medium

ABSTRACT

Methods for storing on a storage or memory medium, and retrieving, and displaying of multiple images in a registered manner, the images have been recorded concurrently. The images may comprise at least 2 video programs. A camera system for recording multiple concurrent images is also disclosed. Lenses and corresponding image sensors are calibrated to have calibrated and associated settings for recording multiple images that are substantially registered images. A registered image may be displayed on a single display. It may also be displayed on multiple displays. A camera for recording and displaying registered multiple images may be part of a mobile phone.

STATEMENT OF RELATED CASES

This patent is a continuation of U.S. patent application Ser. No.12/435,624 filed on May 5, 2009 which claims the benefit of U.S.Provisional Patent Application No. 61/054,290, filed on May 19, 2008, ofU.S. Provisional Patent Application No. 61/055,272, filed May 22, 2008,and of U.S. Provisional Patent Application No. 61/089,727, filed Aug.18, 2008, all of which are incorporated herein by reference in theirentirety.

BACKGROUND OF THE INVENTION

The present invention relates to digital media players. Morespecifically it relates to media players that can play two or moredigital files at the same time and to cameras as the origin of the twoor more digital files.

Digital storage media such as CDs, DVDs, magnetic disks, flash memoryand other storage media which are used for storing and playing forinstance multi-media files representing music and video are known. Onecan use the storage media in devices such as DVD players or computers orpersonal digital devices for watching a movie or video program forinstance. One may also play music recordings. Another playing medium isa flash memory such as used in MP3 players. Another medium may be amagnetic disk.

Sometimes it may be desirable to play several files at the same time. Aknown application is, for instance, an entertainment center as used forinstance in an airplane. Up to 10 or more different video channels mayneed to be provided at the same time, available for selection by usersof a system. In general, the requirement for different signals is solvedby playing different DVDs or optical disks or other media and making thesignals coming from these individual media available in a selectablemanner through a communication channel.

Another application may be providing a movie with different screens or asingle screen that show what is going on at different locations of ascene viewed from different cameras at the same time.

All the above and other applications that are fully contemplated requirea recording and later playback from a single medium of different moviesor files at virtually the same time. In line with current trends, thesevideo and other multi-media files may be high definition.

Accordingly, methods or apparatus providing novel and improved recordingand playback of multiple files at virtually the same time from a singlestorage medium are required.

SUMMARY OF THE INVENTION

One aspect of the present invention presents novel methods and systemsfor recording, storing and concurrent displaying of a plurality of videoprograms.

In accordance with another aspect of the present invention a system isprovided for displaying a plurality of video programs, comprising amultiplexer for creating a signal of time division multiplexed signalsfrom the plurality of video programs, a storage medium for storing thesignal, a player for reading the stored signal, the player containing ademultiplexer for de-multiplexing the signal into a plurality ofdemultiplexed signals, and a display for displaying one of the pluralityof video programs reconstructed from a demultiplexed signal.

In accordance with a further aspect of the present invention, a systemfor displaying a plurality of video programs is provided wherein each ofthe plurality of video programs is a recording of a scene takenconcurrently with each other.

In accordance with another aspect of the present invention, a system isprovided, wherein the storage medium is an optical disk.

In accordance with a further aspect of the present invention, a systemfor displaying a plurality of video programs is provided, wherein acamera is an integrated camera system having at least 2 video imagesensors.

In accordance with a further aspect of the present invention, a systemfor displaying a plurality of video programs is provided, wherein anoptical disk can store a plurality of symbols each having one of threeor more states.

In accordance with a further aspect of the present invention, a systemfor displaying a plurality of video programs is provided, wherein ademultiplexer can process symbols having one of 3 or more states.

In accordance with another aspect of the present invention, a method fordisplaying a plurality of video programs is provided, comprisingmultiplexing a plurality of signals each signal representing one of theplurality of video programs into a multiplexed signal, storing themultiplexed signal in a contiguous manner in a memory or on a storagedevice, reading the multiplexed signal from the memory or storagedevice, demultiplexing the multiplexed signal and creating a pluralityof playable signals, each playable signal representing one of theplurality of video programs, and playing a playable signal on a videodisplay.

In accordance with a further aspect of the present invention, methodsand systems are provided for creating combined and registered imagescreated from 2, or 3 or more cameras, the cameras being either a staticimage camera or a video camera. Also a plurality of cameras may be used.

In accordance with a further aspect of the present invention, a cameramay be a mobile computing device, enabled to communicate wirelessly andhaving at least two lenses from which a combined and registered imagemay be formed. The camera may be applied for photographs and/or forvideo images.

In accordance with an aspect of the present invention, methods andsystems are provided for recording concurrently a plurality of imagesand for displaying the concurrently recorded images in a registeredmanner.

In accordance with a further aspect of the present invention, a methodis provided for displaying at least a first and a second concurrentlyrecorded image in a registered manner, comprising storing in a memorydata representing a first setting of a first lens in a camera forrecording the first image under a first condition, storing in the memorydata representing a first setting of a second lens in the camera forrecording the second image under the first condition, associating thefirst setting of the second lens with the first setting of the firstlens, applying the first setting of the first lens to the first lens,retrieving by a controller from the memory data representing the settingof the second lens associated with the first setting of the first lensand putting the second lens in the retrieved setting; and storing in animage storage device data representing the first image taken through thefirst lens and data representing the second image taken through thesecond lens.

In accordance with yet a further aspect of the present invention, amethod is provided, further comprising storing data representing a mergeline in the first and the second image.

In accordance with yet a further aspect of the present invention, amethod is provided, wherein the data representing the first image dataare data determined by the merge line.

In accordance with yet a further aspect of the present invention, amethod is provided, wherein a setting of a lens is selected from a groupconsisting of focus, zoom, diaphragm, shutter speed, and lens position.

In accordance with yet a further aspect of the present invention, amethod is provided, wherein images are video images and furthercomprising multiplexing image data in accordance with a sampling theoremand storing the image data in a contiguous manner.

In accordance with yet a further aspect of the present invention, amethod is provided, wherein the image storage device is selected fromthe group consisting of a binary electronic memory, a rotating binarydata storage medium, an n-state electronic memory with n>2 and ann-state rotating n-state storage medium.

In accordance with yet a further aspect of the present invention, amethod is provided, further comprising reading the image data from theimage storage device, demultiplexing the image data into datarepresenting at least a first and a second concurrent and registeredimage and displaying concurrently the at least first and secondregistered image.

In accordance with yet a further aspect of the present invention, amethod is provided, further comprising processing the data of the atleast first and second image for display on a single display as a singleregistered image.

In accordance with yet a further aspect of the present invention, amethod is provided, further comprising processing the data of the atleast first and second image for display on at least a first and seconddisplay to provide a registered image of a scene recorded by the atleast first and second image.

In accordance with yet a further aspect of the present invention, amethod is provided, wherein the camera is part of a mobile phone.

In accordance with another aspect of the present invention, a system isprovided for displaying at least a first and a second concurrentlyrecorded image in a registered manner, comprising an instruction memoryholding data representing instructions, a processor for retrieving fromthe memory data representing instructions and executing the instructionsfor performing the steps of: storing in a memory data representing afirst setting of a first lens in a camera for recording the first imageunder a first condition, storing in the memory data representing a firstsetting of a second lens in the camera for recording the second imageunder the first condition, associating the first setting of the secondlens with the first setting of the first lens, applying the firstsetting of the first lens to the first lens, retrieving by a controllerfrom the memory data representing the setting of the second lensassociated with the first setting of the first lens and putting thesecond lens in the retrieved setting and storing in an image storagedevice data representing the first image taken through the first lensand data representing the second image taken through the second lens.

In accordance with yet another aspect of the present invention, a systemis provided, further comprising instructions for storing datarepresenting a merge line in the first and the second image.

In accordance with yet another aspect of the present invention, a systemis provided, wherein the data representing the first image data are datadetermined by the merge line.

In accordance with yet another aspect of the present invention, a systemis provided, wherein a setting of a lens is selected from a groupconsisting of focus, zoom, diaphragm, shutter speed, and lens position.

In accordance with yet another aspect of the present invention, a systemis provided, wherein images are video images and further comprisingmultiplexing image data in accordance with a sampling theorem, andstoring the image data in a contiguous manner.

In accordance with yet another aspect of the present invention, a systemis provided, wherein the image storage device is selected from the groupconsisting of a binary electronic memory, a rotating binary data storagemedium, an n-state electronic memory with n>2 and an n-state rotatingn-state storage medium.

In accordance with yet another aspect of the present invention, a systemis provided, further comprising instructions for performing the steps ofreading the image data from the image storage device, demultiplexing theimage data into data representing at least a first and a secondconcurrent and registered image, and displaying concurrently the atleast first and second registered image.

In accordance with yet another aspect of the present invention, a systemis provided, further comprising instructions for processing the data ofthe at least first and second image for display on a single display as asingle registered image.

In accordance with yet another aspect of the present invention, a systemis provided, further comprising instructions for processing the data ofthe at least first and second image for display on at least a first andsecond display to provide a registered image of a scene recorded by theat least first and second image.

In accordance with yet another aspect of the present invention, a systemis provided, wherein the camera is part of a mobile phone.

DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram of a known way to create a video program;

FIG. 2 illustrates a known way to play a video program;

FIG. 3 illustrates the creation of a multiplexed signal on a storagemedium in accordance with an aspect of the present invention;

FIG. 3 a is a diagram of a two possible frames of a multiplexed signal;

FIG. 4 illustrates the creation of a multiplexed signal on a storagemedium in accordance with another aspect of the present invention;

FIG. 5 illustrates the demultiplexing of a multiplexed signal on astorage medium in accordance with an aspect of the present invention;

FIG. 6 illustrates the demultiplexing of a multiplexed signal on astorage medium in accordance with another aspect of the presentinvention;

FIG. 7 illustrates time reduction in multiplexed signals;

FIG. 8 illustrates a camera system and a display system in accordancewith aspects of the present invention;

FIGS. 9-11 are diagrams of a system in accordance with different aspectsof the present invention;

FIGS. 12-13 are diagrams of displays in accordance with differentaspects of the present invention;

FIGS. 14-15 illustrate multiplexing in accordance with different aspectsof the present invention;

FIG. 16 shows a diagram of an n-valued individually controlled switch;

FIG. 17 is a diagram of part of an implementation of an n-valuedindividually controlled switch;

FIG. 18 is a diagram of a multiplexer;

FIG. 19 is a diagram of a multiplexer in accordance with an aspect ofthe present invention;

FIG. 20 is a diagram of a multiplexed signal in accordance with anaspect of the present invention;

FIG. 21 is a diagram of a buffer;

FIG. 22 is a diagram of a demultiplexer in accordance with an aspect ofthe present invention;

FIGS. 23-24 are diagrams of demultiplexing systems in accordance withaspects of the present invention;

FIG. 25 is a diagram of a system in accordance with an aspect of thepresent invention;

FIG. 26 is a diagram of a user interface in accordance with an aspect ofthe present invention;

FIG. 27 is a diagram of a system in accordance with an aspect of thepresent invention;

FIG. 28 is a diagram of a system in accordance with an aspect of thepresent invention;

FIG. 29 is a diagram of a system in accordance with an aspect of thepresent invention;

FIG. 30 is a diagram of a known video camera;

FIG. 31 is a diagram of a camera in accordance with an aspect of thepresent invention;

FIG. 32 is a diagram of a camera in accordance with another aspect ofthe present invention;

FIGS. 33 and 34 are diagrams of a camera in accordance with yet anotheraspect of the present invention;

FIG. 35 is a diagram of a system in accordance with an aspect of thepresent invention;

FIG. 36 is a diagram of a camera system in accordance with an aspect ofthe present invention;

FIG. 37 is a diagram of a camera system in accordance with anotheraspect of the present invention;

FIG. 38 is a diagram of a surround video display system in accordancewith an aspect of the present invention;

FIG. 39 is a diagram of a multi-program video playing system inaccordance with an aspect of the present invention;

FIGS. 40 and 41 each provide a diagram of a video camera in accordancewith one or more aspects of the present invention;

FIGS. 42 a, 42 b and 43 each provide a diagram of a video display systemin accordance with one or more aspects of the present invention;

FIGS. 44 and 45 provide diagrams of a video image format in accordancewith one or more aspects of the present invention;

FIGS. 46 and 47 each provide a diagram of a video system in accordancewith one or more aspects of the present invention;

FIGS. 48-50 each provide a diagram of a mobile computing device enabledto communicate wirelessly and having two or more lenses in accordancewith one or more aspects of the present invention;

FIG. 51 shows a diagram of two images taken by two lenses in a camera inaccordance with one or more aspects of the present invention;

FIG. 52 shows a diagram of a single image created from 2 separate imagesin accordance with one or more aspects of the present invention; and

FIGS. 53-55 illustrate in diagram calibration settings of lenses inaccordance with further aspects of the present invention.

DESCRIPTION

The storage of a digital image such as a static image or a video programon a storage medium is known. A video program may contain a black/whiteor color video signal, audio signals (mono, stereo or a plurality ofaudio signals), electronic signals which may include but is not limitedto subtitles, date and time information or any other requiredinformation. The video and/or audio signals may be in analog form, theymay be in analog form and being converted to digital signals, which arecommonly binary signals. These signals may also be digital signalsprovided by camera and/or audio equipment. These signals are usuallycombined in a frame and stored on a storage medium such as an opticaldisk such as a DVD disk. Intermediate steps may include signalcompression, for instance according to known compression techniques suchas the MPEG 2 standard. Certain modulation techniques and codingtechniques may also be applied, such as error correcting coding,scrambling, and modulation techniques such as Pulse Code Modulation,Pulse Amplitude Modulation, Pulse Position Modulation, QAM coding or anyother modulation technique. Signals may be added, for instance, frameand synchronization information or any other information that isrequired to store the signal on a storage medium and that allows thecomplete information to be recovered.

In order to allow appropriate retrieval and display or playing of thevideo, audio and electronic signals using one pick-up mechanism such asa laser in the illustrative example of an optical disk, all signals aresampled at certain intervals and combined in a sample frame that mayconsist of interleaved of multiplexed signals. When the sampling ofsignals and their recovery and playing are in accordance with the knownsampling theorem, then a replay signal may be formed from the samplesthat appear to be a good or perfect copy of the original signal. Thesampling of signals and creating a combined serialized frame of video,audio, and/or electronic samples is called serializing, interleaving ormultiplexing of a signal. An example of creating such an interleavedvideo program signal can be stored on for instance a video disk isprovided in U.S. Pat. No. 4,782,402, issued on Nov. 1, 1988 to Kanamaruwhich is incorporated herein by reference in its entirety.

In a player of a video program from a storage medium a signal will beread from, for instance, an optical or magnetic disk or mass memory forinstance as a series of digital symbols. These serialized symbols areusually set in a frame, having samples of a video signal and possiblyaudio signals and other electronic signals. A player has to deserializeor recover the stream of symbols in accordance with the proper samples,apply any required demodulation, decoding or decompression as requiredin an order that is required, and re-constitute a presentation of theaudio and video signals that is appropriate for a user. Deserializing isknown. A description may be found in for instance in U.S. Pat. No.6,282,320, issued on Aug. 28, 2001 to Hasegawa et al. which isincorporated herein by reference in its entirety. Digital signalprocessing, including for use on DVD disks is provided in U.S. Pat. No.6,574,417, issued on Jun. 3, 2003 to Lin et al. which is incorporatedherein by reference in its entirety.

Information streams may be formed from different sources. Differentelectronic sources and formats including video broadcast formats aredescribed in U.S. Pat. No. 7,020,888, issued on Mar. 28, 2006 toReynolds et al. which is incorporated herein by reference in itsentirety.

There are several known ways to code a video signal into a digitalsignal which is generally a binary signal. Presently, a video signal maybe generated from an optical image sensor of which a full video screencomprises a matrix of pixels or pixel elements. Each pixel is coded forinstance according to a palette, a RGB, a CMYK or any other color schemethat is appropriate. A frame or a screen, which may be interlaced, hasfor instance 1920×1080 color pixels which are displayed at a rate ofabout 25 or 30 or 50 or 60 screens or frames per second.

A number of pixels may determine a resolution of a video signal. Theframe rate determines how humans perceive individual static images as afull moving image. In order to perceive a series of discrete imagescomprised of pixels as a movie each 1/24th second a full frame may haveto be shown. This also means that between providing two consecutiveframes there is at least 1/24 second available to do other things. Itdoes not mean that the 1/24 second has to be used to transmit the fullframe. Theoretically, that can be done in for instance a 1/10,000 of asecond. This principle of assigning time slots to a sample of a signal,which may be a video signal is known as Time Division Multiplex or TDM.The replaying of a signal without significant loss of information isexplained by the known sampling theorem, which states that a samplingfrequency should be twice the highest frequency of a signal that onewants to capture in samples. This usually leads to sampling by forinstance pulse amplitude modulation. An amplitude of a sample may becoded in for instance a word of binary symbols or of n-valued symbolshaving one of 3 or more states.

Storage media for video signals are known, for instance as DVD disk,magnetic disk and a mass memory For a DVD disk a video signal is sampledand provided as a stream of usually binary signals, it is multiplexedwith additional binary audio and possibly other signals and written on amedium such as an optical DVD disk. A combined multiplexed signal may becalled a program. A DVD disk may be played by a DVD player where thesignal is read from the disk where video and audio signals are separatedand provided to their respective players. It is necessary to keepsignals synchronized. While a DVD disk is mentioned as a storage medium,a magnetic disk, a tape or a memory such as flash memory or RAM or ROMmemory or any other medium that can store a large amount of data canserve as a video program storage medium.

A video program usually contains two components: a video signal and anaudio signal. An audio signal may be one or more channels of audio.Increasingly, a third electronic signal is added, which may containdisplayable information, such as for instance a subtitle that may besynchronous with the audio. Other electronic information may also beadded, such as audio commentary, menu information, track information orany other information that one may want to combine with a video signal.For all these signals the sampling theorem may apply.

In general, one would like to display all information in a manner thatis synchronous to the video signal. So, if a person is seen speakinginside a video image one would like to have the audio being synchronous.One would also like subtitles to be synchronous to events in thepicture.

One way to assure that all signals will be available at the appropriatetime is to digitize the signals and multiplex the signals in TDM andwrite the composite TDM signal to the storage medium. This is shown indiagram in FIG. 1. A camera 101 provides a video signal. A microphone102 provides an audio signal. And a signal generator 103 provides anelectronic signal that represents, for instance, subtitles. Each unit101, 102 and 103 may provide more than one signal. For instance, camera101 may be 2 or more cameras which may generate a composite image. Acamera may be a video camera. It may also be a thermal sensor or anyother sensor that can register moving images. An audio signal may be asingle channel. It may also be a stereo signal. It may also be asurround sound signal having more than 2 channels. The electronic signalmay also contain a plurality of information channel such as subtitlesand menus. The purpose is to create at least a single serialized signalthat can be written on a storage medium.

The terms multiplex, multiplexing, multiplexer, demultiplexed,demultiplexing and demultiplexer all refer to time division multiplexingor time division demultiplexing unless specifically being identified asnot being so.

All apparatuses provide signals to a coding/multiplex unit 104. Eachsignal may already be a digital signal. If a signal is analog it will bedigitized by 104. The unit 104 may be provided with an external signal109 that contains at least one clock signal to which the processing ofall signals will be synchronized. The clock signal is provided to thecoding/multiplexing unit to control the multiplexer. The multiplexerassures that digital signals representing a signal sample are put in aserialized frame for an appropriate time slot. After passing of thetime-slot (which may be determined by the clock signal) the multiplexerswitches to another signal for inserting its sample into a serializeddigital signal 105. The unit 104 in general, and as is known in the art,may provide other functions, such as: compression of for instance thevideo signal according to a known scheme such as MPEG-2. The unit 104may also scramble the signal. It may also provide a block of digitalsignals with error correction capabilities. Furthermore, the multiplexermay insert synchronization symbols and other symbols that are used forcorrect recovery of the multiplexed signal. After multiplexing andcoding a serialized digital signal is provided to a writing element 106that will write the signal to a medium 107. The writing element mayprovide modulation of the signal. Modulation may also be provided by104.

Ultimately, a video program or part of a video program may be written toa medium 107 that may be played in a player. The writing of a signal maybe done at very low speeds. It may also be done at very high speeds. Thespeed of the writing process may be unrelated to the reading or playingof a signal from the medium. The reading speed, if it is used fordisplaying a video program, may again determined by the samplingtheorem. The process of multiplexing is also known as interleaving.There are different known interleaving schemes.

FIG. 2 shows that a medium which contains a serialized signalrepresenting a program or part of a program can be played by a player. Apick-up or reader 202 reads a serialized signal from the medium 201. Thepick-up unit may provide additional functionality such as amplificationand demodulation. However, these functions may also be provided by aunit 204 that may be a decoder/de-multiplexer unit 204. The pick-upprovides a signal 203 to the decoder/de-multiplexer 204 where the signalmay be error corrected or other wise decoded and descrambled ifrequired. The de-multiplexer has the capability to recognize positionsin the serialized signal, for instance based on synchronizationinformation that was inserted. The de-multiplexer then passes thecorrect samples for the correct display to the correct output channels.The de-multiplexer may work off an external signal 209 which may containa clock signal. A clock signal may also be derived from the signal 203.

Other steps may be involved in playing a sampled signal. One such stepmay be Digital/Analog conversion (D/A). One other step may be low passfiltering. These and other steps are generally known and need noexplanation herein.

The decoder/de-multiplexer then provides the correct video signals to avideo display 205. Audio signals may be provided to one or more audiospeakers 207. Correct signals such as subtitles may be inserted on thedisplay for instance at a position 206.

Variations on the above scheme are known. However, all have the sameapproach that a plurality of signals representing different aspects of aprogram are digitized and combined into a serialized signal on a storagemedium. The serialized signal for playback is read from the medium,de-serialized or de-multiplexed and its component signals are providedto the correct display. Video signals are de-compressed if they werecompressed. The required D/A conversion may take place in 204 or in thedisplay apparatus.

The constraint in correct playback is that the samples have to beoutputted to a D/A converter with the correct speed. This usuallytranslates back to that the storage medium has to be read at a definedspeed. For a DVD disk the reading speed may be an equivalent bitrate ofabout 10 Mbit/sec. For a HDTV disk the reading speed may be about 30Mbit/sec.

For correct playback the sampling theorem determines how many samplesper second should be generated. The resolution of a signal (and thus thecoding of a pixel) determines how much information or bits a samplecontains. The sampling rate is related to the highest frequency that onestill wants to recover without aliasing. This frequency is known as theNyquist frequency or Nyquist rate which should at least be twice thehighest frequency component of the signal that is sampled.

The duration of a program on a storage medium nay thus be a compromisedetermined by the storage capacity of the disk, the required readingspeed and the resolution of the signal. A video program may be around120 minutes, including extras. A disk has a certain capacity, say 4Gigabytes. This determines about the quality that can be achieved if onecan read at a speed of about 10 Mbit/sec.

One may increase the storage capacity of a medium such as a DVD disk,for instance by using smaller features and by using a blue laser. Byincreasing the reading speed to 30 Mbit/sec one may increase the qualityof the signal to HDTV quality.

Currently the trend is to create higher capacity media with higherreading speeds in order to provide higher quality display like HDTVquality.

Higher capacity disk allows for storage of more programs on a disk.However, these programs are usually stored and read in a sequentialmanner. That means that for instance two or more video programs may bestored on a storage medium. However, in general, the two video programsare not stored in an interleaved manner. Furthermore, a program is readfrom a disk with a speed that is set for displaying a single program. Toplay a second program the reading mechanism has to go to the storagelocation (or address) of the second program. Without a second readingmechanism it is physically impossible to read two or more programs fromsubstantially different locations on a medium without some interruptionfor switching of addresses or locations. While electronically this maybe faster than mechanically, it still requires usually too much time tobe able to show different programs at the same time within the bounds ofthe sampling theorem to provide a plurality of High Definition programsat the same time.

In accordance with one aspect a method and a system will be providedthat displays two or more video programs concurrently (including videoand sound and potentially informational electronic signals) by reading asignal from a storage medium. A storage medium may be a magnetic harddisk, a magnetic tape, an optical disk, a memory based medium usingmemory cells such as flash memory, or any other medium that can storemassive amounts of data and the data can be read from a medium at apre-determined speed.

The method and system in accordance with a further aspect of the presentinvention, applies Time Division Multiplexing and multiplexing ofdifferent program signals into at least one sequential signal, whichcontains signals from different programs. at least two programs beingvideo programs.

It should be clear that the recording of a TDM signal may be done at anyspeed of recording. However, the playback of the signals for replay atproper speed and quality should comply with the sampling theorem. For avideo signal that is recovered from a composite TDM signal that meansthat at what is considered common display speed a new video frame shouldbe available each 1/24th or 1/25 or 1/30^(th) or 1/50^(th) or 1/60^(th)of a second, or any other appropriate speed depending what kind ofdisplay method is applied. One may record a screen frame for display athigher speeds for instance, for making a displayed video signal appearto be stable and flicker free. However, other means may also be appliedto make a video display to be flicker free. For instance, one may buffera complete single screen for a single frame period. Usually interlacingeffects affect the noticing of flicker in a video display.

It will be assumed herein that a video program will be stored in such away that it can be recovered and played at least acceptable displayframe speed.

An illustrative example of one aspect of the present invention isprovided in FIG. 3. Herein, 3 video programs are generated using, forinstance, the method or system as shown in FIG. 1. It is to beunderstood that there can also be 2 programs generated. There can alsobe more than 3 programs generated. FIG. 3 shows three program signals310, 302 and 303 being made available. These signals may be digitalsignals. These signals may be ready or almost ready to be written to astorage medium or a memory. The signals may also be an analog signal,which are required to be digitized. The signals 301, 302 and will beprovided to a multiplexer 303. Assume that each of the video programsignals represents an information transfer rate of for instance 10Mbit/sec, which is about the transfer rate for a video program on a DVDdisk.

The sampling theorem requires that in a time slot which transfers atleast one program frame for instance per ‘1/a’ sec wherein ‘a’ may be24, after being multiplexed now requires to contain to contain at least3 (different) program frames. This requires at least an outgoingtransfer rate of 30 Mbit/sec in the example of FIG. 3. The rate willprobably be higher as the multiplexer may add some house-keepingsignals, for instance for synchronization and separation of signalsbelonging to different frames. Because one may want to write themultiplexed signal of combined video programs on a storage medium ormemory 306, some form of error correcting coding may be required, whichwill increase the number of symbols in the multiplex signal. One mayalso scramble the multiplexed signal before it is written to storagemedium or memory, for instance with a Linear Feedback Shift Register(LFSR) scrambler.

Because of the multiplexing, (also called serializing or interleaving)one second worth of signal now contains at least 30 Mbits, while anindividual video program has about 10 Mbits if the video program is notin High Definition. Accordingly, the multiplexed signal has at least 3times as many bits. This has as a consequence that the 1 second ofindividual program now only has at most ⅓ of a second available. Thisthen means that the signal of an individual video program has to bereshaped to fit in its allotted time slot. This may have as aconsequence that pulses have to be narrowed to make room for theadditional pulses. The pulse compression is demonstrated in FIG. 7.

FIG. 7 shows a diagram 700 of a series of binary pulses representing [10 1 0 1 0 0 1 1 0 1 0 1 1 1] in a time coordinate. FIG. 7 shows in 701 acurve representing the same binary data represented in a signal usingthe same time coordinate. The duration of each pulse in 701 has beenreduced to about ⅓ of its original duration in 700. It is easy to seethat enough room has been created to insert additional pulses.

It is not necessary to interleave whole frames, for instance by firstinserting frame 1 of signal 301 of FIG. 3, followed by frame 1 of signal302 and then by frame 1 of signal 303, followed by frame 2 of eachsignal, etc. One may interleave part of a frame, even up to theindividual bits as long as one frame per required frame time can bereconstructed.

This preparing the individual signals for time multiplexing, which mayinvolve buffering, pulse shaping and releasing of the pulses at theright time is done by pulse management units 307, 308 and 309 for eachsignal 301, 302 and 303. If provided, signals are analog signals theunits may also take care of A/D conversion. The pulse managementfunction may also be performed by the multiplexer unit 310. Themultiplexer switch 311 may sample the pulses provided at the right timeat the output of units 307, 308 and 309. These units 307, 308 and 309may be part of a coding/multiplexing unit 300. It should be clear thattiming of the signals and pulses is important. An external clock signal304 is provided to assist in controlling the timing.

The multiplexer is schematically shown inside 300. Each pulse managementunit may provide a pulse already in the correct timing slot. When theswitch 311 is enabled for an output of a pulse management unit, a pulse(or the absence thereof) is connected to the multiplexer unit 310. Theswitch may remain in a position for one or more pulses. After that timeit switches to the output of the next pulse management unit, etc. Thisthen creates a serialized series of pulses inside the multiplexing unit310.

The multiplex switch 311 is depicted in such a way that it may look likea mechanical switch. A binary multiplex switch is known and in generalis an electronic switch that achieves extremely fast switching speeds.

The unit 310 may add additional bits for synchronization and it mayperform an error correcting coding such as a Reed Solomon code andprovide for instance scrambling of the signal. Other services may alsobe provided. For instance, pulse shaping and coding to alleviate effectsof inter-symbol-interference. The unit outputs a serialized binarysignal that is provided to a writing unit 305. The writing unit 305 mayfor instance create an optical signal that is written to an opticalstorage disk 306. The writing unit may also create a magnetic or anelectro-magnetic signal that is written to a magnetic disk 306. Thewriting unit 305 may also be a digital memory writing unit that writesbinary elements to a memory unit 306. The writing unit 305 may be anywriting unit that modulates or modifies a signal to write it to astorage medium or memory 306.

The writing unit 305 and storage/memory medium 306 may be in differentlocations and may be connected wirelessly. In that case 305 may includea modulator, a transmitter and possibly an antenna. Memory/storagemedium or unit 306 in that case may also have an antenna, a receiver anda demodulator, before writing a serialized signal comprising a multipleof programs, including a video program, onto a storage or memory medium.

For instance, 306 may be a mobile phone or a mobile computing devicewhich receives a serialized signal that contains a multiple of videoprograms (at least two) which will be at least partly stored.

Because the signals 301, 302 and 303 are generated real-time by at leasta camera, the system of FIG. 3 has to work real-time. One way thissystem can work slower than real-time is if there are significantbuffers inside units 307, 308 and 309.

The signals 301, 302 and 303 may reach the processing unit 300 through awired connection. The signals may reach the processing unit also througha wireless connection. In that case, a wireless receiver and, ifnecessary, decoder may be assumed to be included with 300, even if it isnot shown as a separate unit.

A system like FIG. 3 may be used if signals are generated andmultiplexed in real-time. It may also be that a signal is created fromother stored signals. This is shown in FIG. 4. This system forillustrative purposes has 3 sources 413, 414 and 415 with stored digitalinformation, which may be video programs. A source may be a storagemedium or a memory. Each source has a reader in this case 418, 416 and417 which generates a signal, 401, 402 and 403 respectively. Such agenerated signal is preferably a digital signal. However, it may also bean analog signal which may undergo an Analog/Digital (A/D) conversion ina multiplex/coding unit 400. Signals 401, 402 and 403 are provided torespective pulse management units 407, 408 and 409 as part of 400. Thesignals are shaped and multiplexed by a switch or multiplexer 411 andprocessed by a multiplexing/coding unit 410. An external clock signal404 may also be provided. A serialized signal is then provided to awriter 405 and a serialized signal is stored on 406. Basically, the sameconfigurations, connections and implementations as in the system of FIG.3 are provided as an aspect of the present invention. The system of FIG.4 also has the capability to multiplex, code and write not in real-time.The multiplexing/coding/writing process may happen slower or it mayhappen faster. Because the signals that need to be multiplexed areavailable in stored form, the slowing down or speeding up is now only amatter of synchronizing all components with each other.

FIG. 3 a. is a diagram of a possible time division multiplex frame.Complying with the sampling theorem and the Nyquist sampling speed,digital representations of signal samples are interleaved and organizedin a frame. For instance, a frame may be a frame 320. Herein, 321 may bea representation of a sample of signal 301 in FIG. 3; 322 may be arepresentation of a sample of signal 302 in FIG. 3; 323 may be arepresentation of a sample of signal 303 in FIGS. 3; and 324 may be arepresentation of another sample of signal 301 in FIG. 3. Frame 320 inFIG. 3 a only shows digital representations of samples of the signals.Frame 330 in FIG. 3 a also shows marks 325, 326 and 327 inserted betweenthe samples. These may be helpful during for instance demultiplexing.Different frames and coding methods for time division multiplexing arepossible and are fully contemplated.

FIG. 5 shows a diagram of a player in accordance with an aspect of thepresent invention. A multiplexed or serialized signal is stored on areadable storage medium or memory 506. This may be an optical disk, amagnetic disk, a magnetic tape, a magneto-optical tape, a memory devicesuch as a mass memory such as a flash memory. Medium 506 may be anystorage or memory device that is readable and can store a multiplexed orserialized signal which contains 2 or more programs, which may be audioprograms, video programs, games, text, images or any program that isindividually playable on a player. In the example of FIG. 5 threemultiplexed programs are stored on the medium 506.

The medium 506 is read by a reader 505. It should be clear that thereader should be able to read the medium at a speed that complies withthe sampling theorem if the video signals are to be displayed. If thethree serialized programs are video programs and the medium is forinstance an optical disk, then the reading speed or at least thetransfer rate from medium to decoder/demultiplexer 500 should be atleast 3 times as fast as the transfer speed of an individual program, ifone wants to play the programs in real-time.

If an individual program requires a transfer rate of 10 Mbit/sec then incase of 3 similar programs multiplexed in a serialized signal mayrequire a transfer speed of at least 30 Mbit/sec. This higher transferspeed translates in the case of an optical disk in a faster “emptying”of the optical disk of information.

The reader 505 provides a signal to a receiver unit 510 that mayreceive, demodulate, amplify, reconstruct, error correcting decode anddescramble the incoming signal. It may do one, some or all of theseactivities. The next step is to demultiplex or deserialize theserialized or multiplexed signal. The demultiplexer may look for asynchronization mark to find a start position, and other marks thatindicate to which frame of program one or more bits belong. Thisinformation, potentially assisted by an external clock signal 504, or bya clock signal that is extracted from for instance the serialized signalwill help control a demultiplexer 511 that may conduct the appropriatesignals belonging to a program to a decoder unit 507, 508 or 509,belonging respectively to a program signal 501, 502 or 503. Decoderunits may be omitted, or may be part of an individual player unit.Decoder units 507, 508 and 509 may also arrange received and multiplexedsignal in their appropriate time slot and make sure that thedemultiplexed bits correctly represent a frame of a program wherein theappropriate series of bits will be recognized as video pixels, audiosamples or electronic signals. Signals 501, 502 and 503 still may bemultiplexed signals, but now containing only samples belonging to oneprogram. Signals 501, 502 and 503 are then provided to respectivereceivers 512, 513 and 514. These are receivers such as DVD playerswhich will extract the correct video, audio and electronic signals. Theymay perform functions such as error correction, decompression and thelike. Each player will then provide an appropriate signal, which may bea video, audio or electronic signal such as a subtitle signal to arespective display 515, 516 and 517. Three lines between player and adisplay indicate a plurality of signals between player and display. Adisplay is assumed to have a video screen, an audio display with one ormore loud speakers and a facility to insert the additional signal suchas subtitles into the video display.

In accordance with an aspect of the present invention, one may now playtwo or more different programs which may be video programs from oneserialized signal, which may be read from an optical disk. It should beclear that the use of three serialized programs is provided as anillustrative example. One may store more than 3 programs on a disk. Forinstance audio programs require much less storage space. A transfer ratefor digital audio is about 1.2 Mbit/sec. Accordingly, one may storeabout 24 multiplexed audio signals on an optical disk that has atransfer rate of 30 Mbit/sec.

It is not required to use an individual player for each deserializedprogram. This is shown in an illustrative example in diagram in FIG. 6.Most components are equivalent to the ones in FIG. 5. The system of FIG.6 in accordance with an aspect of the present invention has a storagemedium 606 with at least 2 multiplexed or serialized programs. In theexample the number 3 is used but the number may also be 4, 5, 6 orhigher. The system has a reader 605 that provides a signal to areceiver/decoder unit 610. The unit 610 provides the signal to ademultiplexing switch 611, possibly assisted by an external clock signal604 and demultiplexes the serialized signal into 3 still multiplexedprogram signals containing only samples related to a specific program.The demultiplexed signal may be provided to a decoder unit forpost-demultiplexing signal clean up in respective decoder units 607, 608and 609, generating digital program signals 601, 602 and 603. Thesesignals are provided to a receiver 612. The receiver 612 has thecapability to extract the correct video, audio and electronic signalsuch as subtitles from each signal 601, 602 or 603 when selected by aselector 613. The receiver 612 may then decode only the selected signaland provide the three signals (video, audio and electronic such assubtitles if present) to a display 614.

There are many variations on the configurations of receivers and numberof displays of which some in accordance with a further aspect of thepresent invention will also be provided.

In one embodiment in accordance with an aspect of the present invention,FIG. 8 shows a configuration wherein as an illustrative example threecameras or three video sensors each having a corresponding lens 801, 802and 803 are used to record a scene from different view points. The threevideo signals are multiplexed and recorded by a coder/multiplexingdevice 804 as provided earlier and the multiplexed signal is stored on astorage or memory medium 805.

The availability of 3 (or more) signals representing a scene viewed fromdifferent positions allows a display of video that offers an immersiveexperience. The medium 805 can be played in a medium player 806, whichprovides a multiplexed signal to a decoder/demultiplexer 807. The device807 provides then 3 demultiplexed program signals, which have to beconverted to a playable format by players 808, 809 and 810. Each of theplayers provides a signal to be displayed respectively by displays 812,813 and 814. This allows a viewer or user 811 to be experiencing animmersive video and/or audio experience.

For illustrative purposes, only 3 signals were multiplexed anddemultiplexed for display. It should be clear that also more (or fewer)signals can be used. For instance, one may create a system wherein aviewer is completely surrounded by displays such as video displays. Forclarity the individual program players are shown as individual units.One may also combine the players with the decoder/demultiplexing unit807, so that practically an immersive video, audio, or audio/videosystem may be provided as an integrated system. One may provide switchesthat switch on/off or selects one or more of the display to display asignal.

In accordance with a further embodiment of the present invention, FIG. 9shows a system using multiple displays 902, 903 and 904. A storage ormemory medium 900 contains a multiplexed signal which is decoded anddemultiplexed by 901. In this case 901 also contains three players toplay a program signal provide the signals to displays 902, 903 and 904.The players may also be provided and controlled individually, though notshown as such, such a configuration is fully contemplated. A viewer oruser may thus view three programs at the same time. These programs maybe related, they may also be unrelated. For instance, a viewer mayreview programs that were recorded at the same time and may try to finda relationship between the programs.

In accordance with a further embodiment of the present invention, FIG.10 shows a diagram of a single display based demultiplexing system.Herein, a storage or memory medium 1000 provides a multiplexed signal toa decoder/demultiplexer 1001. The unit 1001 also includes a programplayer which provides a signal to a display 1002. The program player isprovided with one of the demultiplexed signals based on a selectedprogram. A program may be selected by a viewer/user 1011 by using acontroller 1003. The controller provides a signal to a selector in 1001that provides the selected program to the player and to a display 1002.The unit 1001 makes available all demultiplexed signals, but only theselected signal will be played. The viewer 1011 may switch almostinstantaneously between programs.

In accordance with a further embodiment of the present invention, FIG.11 shows a diagram of a multiple display multiple user demultiplexingsystem. In the system of FIG. 11 also a single storage or memory elementis provided with a multiplexed signal representing at least 2 programs,which may be video programs. In the illustrative example, 3 videoprograms are stored in a multiplexed way on a storage medium 1100. Themultiplexed signal is read from 1100 by a reader 1101. The multiplexedsignal in one embodiment is provided to 3 decoder/demultiplexer/playerunits 1102, 1103 and 1104 for users 1105, 1106 1107 respectively withdisplays 1105, 1106 and 1107. Each of the units 1102, 1103 and 1104 alsocontains a selector allowing to select a program to be played. Thus, thesystem as provided by FIG. 11 allows different users to play one of aset of programs to be played from a single storage medium provided.

In a further embodiment, picture-in-picture (PIP) display is provided.PIP, which may the display of one large video image, with at least onesmaller video image displayed within the larger image. PIP may also bepresented split screen format wherein no overlap of images occurs. Insplit screen one image can be larger than another image. A user mayswitch between images, enlarge a previously smaller image or bring afirst image to a foreground. In one embodiment, one may show two or morepictures on one display screen 1200 as shown in FIG. 12. For instanceimages 1201, 1202 and 1203 may be displayed on a single display. Thismay require that the display has three players or tuners. A processormay allow a user to select one of the displayed images to be enlargedand play in for instance full screen mode. Such an embodiment is shownin FIG. 13 with a display 1300 with a large image 1301 and for instance2 smaller images 1302 and 1303. All images are drawn from a singlemultiplexed signal that was stored on a storage or memory medium.

N-Valued Storage Media

It is known that video signals are increasingly provided in HighDefinition (HD) format. This requires a transfer rate in a storagemedium in the reading process to be about 30 Mbit/sec for propertransfer of one HD program on for instance a DVD disk. This means that 3HD type programs that are time division multiplexed to be read at 90Mbit/sec if one wants to show the 3 HD programs in real-time from thestorage medium. One way to solve this issue is to have the DVD playersignificantly increase its rotating speed. However, one can then store adiminished amount of data per program, if one continues to use the samestorage medium, and assuming that one can increase the playing speed ofa medium.

One way to increase the capacity of a storage or memory medium is tostore symbols that ate not bits but can assume a state of 3 or morestates. Such media are known and fully enabled. Multi-state memorydevices are for instance described in U.S. Pat. No. 7,345,934, toGuterman et al. which is incorporated herein by reference. An n-statelogic way to realize n-state memory is disclosed in U.S. patentapplication Ser. No. 12/061,286, filed on Apr. 2, 2008 which isincorporated herein by reference. Storage media such as optical disksenabled to store for instance 8-level symbols are disclosed in U.S. Pat.No. 7,126,897, to Takeuchi et al., U.S. Pat. No. 7,149,178, to Wong etal. and U.S. Pat. No. 7,136,333, to Wong et al. which are all threeincorporated herein by reference in their entirety.

In accordance with a further aspect of the present invention, a storageand/or memory medium is provided that can store multiplexed videoprograms in multi-valued or n-valued or n-state symbols, an n-valued orn-state symbol is a symbol that can assume one of 3 or more states. Amulti-valued or n-valued or n-state symbol distinguishes itself from abinary or 2-valued symbol also known as a bit which can assume one ofonly 2 states. A symbol herein is represented by a signal. An n-valuedor n-state symbol is then represented by a signal that can occur in ndifferent states with n>2. It also is intended to mean that a signalrepresented one of n states can clearly and unambiguously bedistinguished from any other state not being that state forsubstantially most of the time. Like with any signal noise anddisturbances may influence error free transmission and detection ofsignals. But states can be distinguished from each other. States can berepresented by different physical aspects. For instance a state can berepresented by an amplitude or range of amplitude, of intensity, ofposition of phase, frequency, wavelength or any other physicalphenomenon that may occur in 2 or more states. For an n-state signal asignal can occur in more than 2 states.

By using symbols with more than 2 states one may maintain a relativelylow reading speed while being able to store multiple programs such asvideo programs in multiplexed such as time multiplexed format on astorage or memory medium.

One may, for instance, assume that a HD video signal currently requiresa reading or data transfer speed of 30 Mbit/sec from a HD DVD opticaldisk. Ignoring multiplex and additional coding overhead, one may assumethat a 3 times multiplexed HD program signals then requires a playbackor transfer speed of 90 Mbit/sec to obtain real-time processed andplayable HD program signals. By using, for instance, 8-level symbols forstoring information on a storage medium the symbol transfer rate may bereduced with a factor 3 to 30 MegaSymbols/sec while still maintaining aninformation transfer speed of about 90 Mbit/sec.

The storage/memory component of a system for storing and retrievingn-valued symbols at least two time-multiplexed programs, which may bevideo programs on a storage/memory medium thus has been enabled.

Another component that may need to be implemented in n-state technologyis the multiplexer/demultiplexer component. One maymultiplex/demultiplex and code/decode in different ways. One may performmultiplexing and coding in one embodiment in binary form, followed by aconversion by a converter, such as a Digital/Analog (D/A) converterwhich converts a plurality of bits into a signal which may assume one ofn states with n>2.

The above embodiment is illustrated in FIGS. 14 and 15. FIG. 14 shows adiagram of a multiplexing/coding system. Three binary signals, 1401,1402 and 1403, which may represent video programs are inputted intopulse shaping units 1407, 1408 and 1409 respectively after which amultiplexer switch 1411 which may be under control of a clock signal1404 forms a multiplexed signal which may be inputted into a coder 1410which outputs a binary multiplexed signal. Pulse shaping may also occurelsewhere in the system. Coding may involve error correcting coding. Itmay also involve coding steps which improve inter-symbol interference.Other steps may also be involved in the generating of the actualmultiplexed signal. These steps are known in the art and are assumed butnot specifically shown. The purpose herein is to show steps of timedivision multiplexing without getting lost in the details. This appliesto other diagrams shown herein as well. For instance, a multiplexedsignal may be modulated before either being written or transmitted.These and other steps, if required, are assumed to take place withoutbeing identified or shown in diagram.

The coding unit 1410 outputs a time division multiplexed binary signaland provides the binary signal to a converter 1412 which generates ann-valued signal 1413. The converter 1412 may be a D/A converter.However, the converter 1412 may also be a modulator, such as aQuadrature Amplitude Modulation (QAM) converter. As disclosed in U.S.Pat. No. 6,178,144, to Huber which is incorporated herein by referencein its entirety, a series of bits or a word can be coded for instance ina 256-QAM constellation and written to a magneto-optical medium. Thesignal can also be recovered in its entirety from the medium.

The principle of the multiplexing and conversion is shown in FIG. 15.Coding is ignored in the diagram, though it is pointed out that codingsuch as error-correcting coding, pulse shaping, ISI (inter-symbolinterference) improving coding and other steps are important and known.The same applies to synchronization marks and coding and otherrequirements. Pulse duration and shapes are provided in a diagram inFIG. 15 and may not reflect actual pulse shapes. They are intended toreflect the requirement of slot fitting of a plurality of signals.

In FIG. 15 box 1501 shows 3 parallel binary signals as provided to amultiplexer 1502. It is assumed, for illustrative purposes, that unitsof 6 bits of a channel will be interleaved for 3 channels. It is shownin box 1503 what a multiplexed signal may look like. A time slot thatfirst has 6 bits now has 18 bits, being the multiplexed signal. Eachgroup of 6 bits representing a word of 6 bits of a channel. One may thenprovide the serial signal to a converter 1504. The converter may be aD/A converter that converts a word of 3 bits to a 8-valued signal thatcan assume 1 of 8 discrete states. This is represented in box 1505wherein an 8-valued signal is represented by an amplitude. It should beclear that other representations and signals are possible. As was statedbefore, the converter may for instance be a QAM module. It may also be aFrequency Shift Keying (FSK) module, wherein a binary word is translatedinto a signal having one of at least 3 frequencies, or it may be anyother n-state representation. The conversion unit may thus be any unitthat converts a word of at least 2 bits into a signal of having at least3 states.

One of ordinary skill in the art will be able to recover an n-statesignal by conversion back to a binary signal and by demultiplexing thesignal and thus recover an individual program signal from a storedmultiplexed signal.

A disadvantage of working with binary words representing an n-valuedsignal is that synchronization of the words is required to maintain theintegrity of the n-valued symbols. Even a single bit shift will mostlikely create a signal at recovery that is not the correct signal.

In a further embodiment, it may be advantageous to apply an n-statemultiplexer. In an n-state multiplexer k input signals are provided andare switched to an output so that a time multiplexed signal of k signalsis created at the output. The difference with the binary case being thatall input and output signals or pulses can have one of n states.

Such an n-state multiplexer that reduces k signals to 1 signal is a k: 1n-state multiplexer. One may also create a k/p n-state multiplexerwherein an input of k signals is reduced to p output signals. Such ann-state multiplexer requires an n-state switch that switches an n-stateinput signal to an n-state output signal under certain conditions. Sucha switch is disclosed by the inventor in, for instance, U.S. Pat. No.7,218,144, issued on May 15, 2007 and U.S. Pat. No. 7,355,444, issued onApr. 8, 2008 which are incorporated herein by reference in theirentirety. A related n-valued switch is also disclosed by the inventor inpending U.S. patent application Ser. No. 11/964,507, filed on Dec. 26,2007 which is incorporated herein by reference.

An individually controlled n-state switch is schematically shown in FIG.16 as 1600. Switch 1600 has an input 1601 which can provide a signalhaving in this illustrative example one of 3 states. The 3 possiblestates are 0, 1 and 2. A state may be represented by a voltage. A statemay also be represented by light of a certain wavelength. A signalhaving one of 3 states may be outputted on an output 1603. The switch1600 is under control of a control signal provided on control input1602. The control signal may also have one of 3 states. The numberinside the switch indicates for which state the switch is ‘conducting’.The term conducting in the context of the switch 1600 means that output1603 provides a signal representing the same state as the signalprovided at the input 1601. In actuality, the switch may not ‘conduct’at all in a physical sense. For instance, an input signal may be anelectrical signal and an output signal may be an optical signal, whichmay be transformed back into an electrical signal. The underlined numberinside the circle representing the switch indicates for which state ofthe signal on the control input 1602 the signal provided on output 1603has a state identical to the state of the signal provided on input 1601.In this case that is for state 2. When a control signal on input 1602 isnot 2, the signal provided on output 1603 has the same state as thesignal on input 1601 has when 1601 has absence of signal. Thisdefinition is required because the absence of signal may represent astate.

FIG. 17 shows a diagram for a possible physical implementation of theswitch of FIG. 16. Assume that a state 0 is represented by a signalgenerated by a signal generator 1704 requiring powering by a source1701. The state 1 is represented by a signal generated by a signalgenerator 1705 requiring powering by a source 1702. The state 2 isrepresented by a signal generated by a signal generator 1706 requiringpowering by a source 1703. If none of the generators are powered thereis absence of signal. An output is generated on an output 1707. Switches1708, 1709, 1710, 1711, 1712 and 1713 are used to enable the n-stateswitch. Switches 1708, 1709 and 1710 may be enabled by the same signal,being for instance the control signal representing state 2. Thesecontrol signals are not shown to prevent undue clutter. However, one mayassume that each of the switches 1708, 1709 and 1710 will be connectedto ground if their enabling signal does not represent the state 2. Whenthe control signal for the switches represents the state 2 all threeswitches will close and create a connection with contacts 1714, 1715 and1716. One can see in the diagram of FIG. 17 that this partially enablesthe power circuits.

The switches 1711, 1712 and 1713 are all controlled by the input signalas provided on input 1601 in FIG. 16. However, the switches are enabledby different signal states. Switch 1711 closes when the input signalrepresents state 0. Switch 1712 closes when the input signal representsstate 1. Switch 1713 closes when the input signal represents state 2.Accordingly, only one circuit is closed or all circuits are open. Output1707 thus has a state 0, 1 or 2 or the same state as the input signalwhen the control signal is in state 2; or the output 1707 providesabsence of signal if the control signal does not represent state 2.

One may actually create simplified versions of the switch and these arefully contemplated. The purpose of the schematic of FIG. 17 is to showthat an n-state switch is fully enabled. A simplified switch only hasfewer components and may not serve the purpose of explaining the n-stateswitch.

It is easy to contemplate modifying the switch of FIG. 17 to conductwhen the control signal represents state 1 and the same for when thecontrol signal represents state 0. One can easily expand the shown3-state switch to any value of n>3.

In FIG. 18, an n-state multiplexer 1800 in diagram is shown. Thismultiplexer has 3 inputs 1801, 1802 and 1803 each providing for instancesignals having one of n states. A control input 1806 provides a k-statesignal that control which of the inputs is ‘conducted’ to the output. Ifrequired, the control input 1806 may be a plurality of inputs. Therequirement is that 1806 can control k individual switches. In thisexample, k=3. The signal as provided in FIG. 15 is an 8-valued signal.Clearly, an 8-valued signal can control 3 individual 8-valued switches.However, one may want to multiplex in some cases 9 or more inputsignals. In that case one may have to use multiple control inputs 1806or a k-valued individual switch with k>8.

One embodiment of a 3:1 multiplexer is shown in FIG. 19. Three inputsignals are provided on inputs 1901, 1902 and 1903 to individuallycontrolled n-state switches 1907, 1908 and 1909. Each of the switches isclosed for a different state of the signal provided on control input1906. Depending on the state of the control signal on 1906 the state ofthe signal on output 1907 has the same state as a signal on one of theinputs 1901, 1902 or 1903. Accordingly, FIG. 19 provides a k:1multiplexer for an n-state signal, with in the illustrative example k=3.

A demultiplexer corresponding to the multiplexer of FIG. 19 works on asimilar principle. If one multiplexes multiple consecutive input pulsesfrom a single input it is required to restore the pulse spacing for theindividual demultiplexed signals. In another embodiment, an n-statemultiplexer is provided that multiplexes a single pulse from eachchannel and then returns to the first channel. The n-state demultiplexerof such an n-state multiplexer has much less of a re-spacing issue asall pulse will be equally distributed over a time slot.

A diagram of demultiplexed n-valued signals which may be n-valued pulsesfor one program is shown in FIG. 20. One pulse 2001 with value s1 isspecifically identified. Other pulses s2, s3, s4, s5 and s6 are alsoprovided. The graph covers to time slots: one

Timing buffers in binary logic are known. They may be known in n-stateor n-valued logic. One may use multiple binary buffers wherein n-statesignals are represented as binary words. An example of an n-state bufferto evenly distribute pulses is provided in FIG. 21. A first buffer 2101is an n-valued shift register with 3 n-valued shift register elementswhich may be created from n-valued memory elements. The demultiplexedpulses are shifted into 2101 on input 2104. The shift register contentis shifted to the right based on a clock signal 2103 derived from thedemultiplexer generating the n-valued pulses of this channel. This meansthat the clock pulses of 2103 will also not be uniformly distributed.After 3 clock pulses the shift register 2101 is full and a second clockpulse 2105 will enable transfer of the content of 2101 to a secondn-valued shift register 2102. The timing should make sure that 2105happens before 2102 needs to be read and before the new pulses s4, s5,and s6 need to be read into 2101.

Occurrence of a clock signal 2107 (being uniform and with a period ⅓ Tin this example) enables reading 2102 in a uniform fashion. At the timethat 2102 is being shifted and read the n-valued shift register 2102 ofs1, 2 and s3 shift register 2101 is being filled with s4, s5 and s6.Every cycle the content of 2101 is moved into 2102, 2102 is read while2101 is being filled. This creates a uniform series of n-valued outputsignals from a non-uniform series of input signals. Other ways ofn-valued buffering are possible and are fully contemplated.

FIG. 22 shows an illustrative example of an n-valued demultiplexer 2200.A multiplexed n-valued signal is provided on 2204 to an input of threeindividually controlled n-valued switches 2207, 2208 and 2209. Allswitches have as control signal an n-valued signal that is provided oncontrol input 2206. Each of the individually controlled n-valuedswitches is enabled by a different state of the control signal. So onlyone of the switches is enabled at any time and provides an output signalon its output, being 2201, 2202 and 2203 respectively. A not enabledswitch may provide absence of

As an illustrative example each TDM time slot is occupied by a singlesignal. This is not a requirement. One may apply cumulativemultiplexing. For instance, one may superimpose optical signals ofdifferent wavelengths, wherein a state is represented by an intensity.In such an embodiment one should preferably not use absence of signal asa state, or at least only for one wavelength.

In accordance with one or more aspects of the present invention, ann-valued multiplexer is provided enabled to time multiplex at least 2n-valued datastreams into a multiplexed n-valued signal with n>2. Beforethe multiplexed signal is written to an n-valued storage or memorymedium the multiplexed signal may be n-valued scrambled and n-valuederror correcting coded. A Reed Solomon (RS) code is multi-valued andshould not require further explanation. Scrambling can take place withfor instance an n-valued Linear Feedback Shift Register (LFSR)scrambler, which has a corresponding descrambler. This aspect isdisclosed by the inventor in U.S. patent application Ser. No.10/935,960, filed on Sep. 8, 2004 which is incorporated herein byreference.

Assume that a High Definition DVD of about 15 GB capacity is required tostore a full HD Video program using compression. Optical disks with thatcapacity are currently available. A disk that can store 3 HD programsthen requires 45 GB storage capacity. The transfer rate for amultiplexed signal of 3 HD video programs is 3 times 30 Mbit/sec=90Mbit/sec. Calimetrics, a firm now defunct has developed at least oneembodiment for n-valued storage media using 8-valued symbols that wouldenable a storage medium being an optical disk with an equivalentcapacity of 200 GB and a transfer rate of 200 Mbit/sec. Such a mediumwould sufficiently enable the storage and reading of at least 3 HD videoprograms to be demultiplexed and enabled to be played concurrently inreal-time.

A multiplexed signal may be provided to a player directly from a storageor memory medium. In a further embodiment, a multiplexed signal may alsobe provided by a transmitter and received by a device such as a mobilecomputing device or a multi-media player. The multiplexed signal may bestored in the device for instance in a mass

In a further embodiment, a player may have access to two or moredatastreams that are available in separate and non-multiplexed form. Forinstance, one may store two or more programs which may be video programson a storage or memory medium. Each program as stored may be stored inplayable form. This is shown in FIG. 23. Herein, three players 2301,2302 and 2303 are shown. Each video program is stored in a memory orstorage medium. For instance, a first program for the first player 2301is stored in a medium or memory 2307. The memory elements or storagelocations of the data are substantially contiguous and are determined bya location or memory address 2306. For playing a program in a playerunder control of a processor 2300, such a processor may initiate anaddress with an address signal 2304 through an address coder 2305 tostart recovering data for providing the data through a player 2308 to adisplay 2309. Especially in media such as magnetic disks, magnetic tapeand an optical disk, transfer of contiguous data is very fast. Thereading element in those storage media does not have to movesubstantially and just follows a fairly easy path. The same applies fora memory. Once a memory is read from an initial address, reading of datafrom contiguous address spaces is fairly easy.

One may provide different programs, which are located at totallydifferent locations or addresses or address space. This is shown asindividual players 2302 and 2303. One may switch between programs,however this requires first of all overhead required by physicallyfinding the new address or storage location. Secondly, if one wants toswitch back and forth between the three programs one has to administerexisting addresses. This can be done by storing the current address andactivating the new address. However, this also creates overhead whichmay interrupt the program. This may require at least three individualplayers that operate fairly independently of each other. Even then,playing from a single disk or a tape or a memory without continuousinterruptive switching effects may not be possible.

In a further embodiment in accordance with an aspect of the presentinvention, a method and system is provided for multiplexing at least twoavailable signals into a multiplexed substantially contiguous signal.This is shown in diagram in FIG. 24. A memory 2403 contains amultiplexed signal representing (in the illustrative example) the dataof 3 concurrent video programs. The data are stored on substantiallycontiguous locations 2402 in a storage medium or contiguous addresses2402 in a memory. An external signal 2400 initiates the playing of themultiplexed signal from an address translated by an address translator2401 from information included in 2400. The memory or storage medium isread and its signal may be conditioned by a signal conditioner 2404. Thesignal conditioner may perform additional tasks such as stripping ofoverhead, error correction and/or descrambling. The signal from 2404 isthen provided to a demultiplexer 2405 which splits the multiplexedsignal in the three individual program signals. As an illustrativeexample, three players 2406, 2407, 2408 are shown. Each playerconditions the received signal for playing by display 2409, 2410 and2411 respectively. It has already been shown that different embodimentsfor display can be provided such as shown in FIGS. 8-13 and described indetail herein. These display embodiments and others may also be appliedin the embodiment of FIG. 24 and are fully contemplated.

In one embodiment, a system is provided that creates a multiplexedsignal from different sources. This is shown in diagram in FIG. 25. Thesystem comprises a computing device 2504 which has access to a memory orstorage device 2505. The device 2504 can be controlled or provided withcommands through 2507. The computing device 2504 also has a multiplexer2509. The device can read 3 or more independent signals from inputs2501, 2502 and 2503. One may also provide 3 or more independent signalssequentially on one of the inputs. In one embodiment three or moreindependent signals are first stored on 2505. The device through acomputer program that is stored on a memory and retrieved and executedby a processor first analyzes the three signals. It determines theamount of data per program, the combined amount of data, it may analyze2505 to establish that enough information is available. Assume that theindividual program signals will be multiplexed according to a fixedlength word of symbols. Assume as an illustrative example that 3 bits orin the n-valued case 1 8-valued symbol per stream or channel will bemultiplexed. After multiplexing a first, a second and a third channelmultiplexing will again start with the first channel. The computerprogram may create frames and superframes for the multiplexed signal andmay prepare insertion of synchronization symbols and other overheadsymbols into the multiplexed signals. The computer program may alsoprovide location information into the multiplexed signal for instancerelative to the beginning of the multiplexed signal. After preparation,the signals are actually multiplexed by multiplexer 2509 andsynchronization and overhead symbols are inserted and a completemultiplexed signal is stored in a memory or any other storage medium2510.

One may keep the signal there for playback. However, in a furtherembodiment the multiplexed data 2506 may be written to a dedicatedstorage medium 2508, which may be a storage medium or a memory. Thispart of the process is the creation and storage or writing of amultiplexed signal. This process does not have to take place in playbackreal-time and may take place at a slower than playback speed, butpreferably in a higher speed than playback. Accordingly, one has now inaccordance with an aspect of the present invention an apparatus and amethod for creating and recording a multiplexed signal containing atleast 2 independent signals which may be video programs. Furthermore,the multiplexed signal may be written to a media that may be played in adifferent apparatus and may provide at least two concurrently playableand displayed programs which may be video programs at a real-time speed.

One may implement the method in a computing device, which may be astandard computer with a processor, memory, a hard drive, one or moreoptical disk players, a USB port for reading data from a flash memory,and at least one optical disk writer and a USB port for writing data toa flash memory. A program is provided for reading two or three signalsto disk, for preparing the hard drive for writing the multiplexedsignal, for actually multiplexing of the signal and completion of themultiplexed signal and writing the multiplexed signal is such a way thatthe signal is written in a contiguous manner on the disk or memory andwill also be transferred as such to a target medium. The target mediummay be written in a contiguous way. Currently, there are tools that willanalyze a hard disk and map the available sectors and reserve contiguoussectors for storing data. Methods thereto are disclosed in U.S. Pat. No.7,280,745 issued on Oct. 9, 2007 which is incorporated herein byreference. Use of proactive defragmenters and other disk defragmentersmay help in creating contiguous data clusters on a hard disk. One wouldlike to have contiguous clusters on a hard disk if one wants to play themultiplexed file in concurrent video programs from the hard disk.Writing of data files on a new or clean memory or storage medium usuallycan take place in a contiguous fashion.

FIG. 26 shows a user interface of a computer program that may initiateand execute creation of a multiplexed signal. A user first has toidentify sources from which signals are to be used for multiplexing. Inthis example up to three sources 2601, 2602 and 2603 can be selected.Enabling the selection will also provide an opportunity to select a fileto be multiplexed. For instance a source_1 may for instance be a file ona hard-disk. Source_2 selected in 2602 may for instance be from a USBport, like a memory stick. Source_3 in 2603 may for instance be a DVD inan optical disk player. However, one may select 2601, 2602 and 2603 asbeing a DVD in an optical disk player. If one has only one optical diskplayer the program will request switching disks after a disk has beenprocessed.

Different sources may be applied and are fully contemplated. Any signalsource that can be recorded will be processed by the system if thesystem is adequately equipped.

The application of which 2600 is the interface may automatically storethe signals from the source. One may also provide a name and a mediumfor the storage files in 2604, 2605 and 2606 as target_1, target_2 andtarget_3 respectively. While not essential for a single multiplexingeffort, one may want to re-use some of the files in the future. In thatcase one has already a “multiplex-ready” file, which may be a source ina future multiplexing. Furthermore, a target determines also the mediumthat the multiplexed signal or file is written to. It should be clearthat writing can be done at a range of writing speeds. It can be doneslower than real-time playing speed. It can be done at faster thanreal-time playing speed. A multiplexer may be a buffer or memory or astorage to which signals are temporarily written and then retrieved tobe put in proper multiplexed order. Each target may require a specificcoding format that will be taken care of in this step. In general, onemay conclude that a multiplexed file is merely a data file withcontiguous data and that it is the final writing to the medium thatdetermines the proper coding. Both multiplex signal preparations arecontemplated and are enabled. Other preparation methods may be possibleand are also contemplated.

The next step in creating a multiplexed signal stored in a single fileand that can be stored in a contiguous fashion after providing the aboveinformation may be an analyzing step. In the analyzing step which may beinitiated by clicking a button ANALYZE 2616, the system determines forinstance how much storage capacity 2607 is required, the playing time2608 which should be the playing time of the longest playing source andif sufficient capacity is available. Other information may also beprovided. For instance, if no sufficient capacity is available, theapplication may provide several suggestions. For instance, it mayrecommend creating 2 or more multiplexed signals. It may offer theopportunity creating a multiplexed signal from only 2 signals instead of3. The system may also make other suggestions. Those suggestions may betechnology related. For instance the system may detect that a sourcevideo format may need to be transformed. It may also be that because ofthe file sizes capacity constraints are important. Like with earlycreation of CD-ROMs in the past, especially from music files, capacityconstraints may initially be important. However, as formats become morestandardized and storage capacity is less constrained, for instance, byapplying n-valued storage media, the analyzing step may be skippedcompletely.

Fields 2610, 2611, 2612 and 2613 can be used to provide names to theprograms which may be video programs which are to be multiplexed. A usermay also provide comments that can be retrieved.

A user may hit 2614 to create a multiplexed signal that will be storedon the hard disk or other mass storage or memory of the system. Thesystem will select the proper methods to find or create if neededcontiguous locations or addresses for storage. It provides all thecorrect overhead in the signal to become playable from the hard disk ormemory and it will write a complete multiplexed signal.

In one embodiment, all three signals may be playable by three playersthat have a common configuration. For instance, the players may all 3 beDVD players that play a video program, an audio/video program or anaudio program. In a further embodiment the signals with a multiplexedsignal are coded for a specific player. Each signal within themultiplexed signal maintains it owns integrity. A demultiplexer thenprovides three signals that have there own integrity. However, it is upto the user to provide the correct player. It may also be that eachsignal within a multiplexed signal is assumed to have a common formatwhich may for instance be one of: video, audio, audio/video, staticimages, text and graphics for instance. In that format it may be assumedthat all players can play any of these formats and automaticallyrecognize the format from the signal and will play accordingly.

The command 2615 ‘write to Target’ writes the multiplexed signal to atarget medium, which may be as stated before any storage or media mediumthat can store the multiplexed signal. For instance, the multiplexedsignal may be written to a removable optical disk or a flash memory. Itcan then be removed from the system and provided to a system forplaying.

FIG. 27 shows a diagram of a possible player of a multiplexed signal. Astorage or memory medium 2701 with a multiplexed signal is read by acomputing device 2702. A user by an input device 2704 may providecontrol commands 2705 to the device 2702 to play one or more of theprograms contained in the multiplexed signal. The multiplexed signal maybe partly or completely stored in a local storage or memory device 2703,controlled by the computing device 2702. The multiplexed signal may beprocessed partly inside the computing device to prepare it fordemultiplexing and playing. When a command for playing is provided themultiplexed signal is provided to a demultiplexer 2706, which providesit to a player 2707 which provides a playable signal which may be avideo program to a display. It is to be understood that a video programmay contain one or all of a video signal, an audio signal and anelectronic signal that may provide additional information such assubtitles and/or menus. A display 2708, 2709 and 2710 may play allcomponents of a video program.

Increasingly, people may play videos on their personal computers orcomputing devices. The availability of computer screens is such thatpeople are using two or more displays with a computing device. The useof two or more displays allows for a more immersive multi-mediaexperience. The display of a surround video using video images from twoor more perspectives played at the same time on different screens mayassist in providing an enhanced multi-media experience.

Different embodiments are contemplated for implementing a player on acomputing device. In a first embodiment, a real-time demultiplexer maybe applied with a device 2707 which will create and conditions thedemultiplexed signals and provides these signals which are ready to beplayed for instance by a DVD player circuit to individual displays 2708,2709 and 2710 which all have their own DVD player circuit. Such playercircuits are known and are commercially available for instance fromcompanies like Broadcom of Irvine, Calif., with the BCM7440 chip or areembedded in products and can be used as individual players.

Chips that contain a demultiplexer and that can provide for instancedual playable video program streams are available. One may expand thosecircuits to 3 or more playable video screens. In such an embodiment onemay consider the demultiplexer 2706 and conditioning device 2707 as asingle unit 2711 which may provide directly 3 playable video signals on3 different outputs which may be inputted to three different displays. Auser may control each of the individual displays for instance acharacteristic such as image contrast or brightness with device 2704through a signal 2712 which may control each of the displays 2608, 2709and 2710 though only one connection to 2710 is drawn in FIG. 27.

In yet another embodiment, one may take advantage of the availability ofhigh-speed digital signal processors (DSP) and availability of fast massmemory. One may thus perform demultiplexing, decoding, conditioning andproviding of playable signals on different outputs by a singleprocessing unit. If the execution speed of a DSP for instance 3different video programs is higher than 3 times the speed of processingan individual video program including the overhead to temporarily storeand retrieve samples and overhead such as storing and retrievinginstruction pointers, the sampling theorem allows a time divisionmultiplexing processing and generating of video program signals.

This is shown in diagram in FIG. 28. A storage or memory device has themultiplexed signal stored in 2801. It shows the multiplexed elements ofthree video programs CH1, CH2 and CH3 for 3 time periods. Themultiplexed signal may contain additional information such assynchronization and error correction information, which if required maybe assumed but are not shown. The frames, signals, overheads and otherelements of multiplexed video program signals are known and available inpublished standards and literature. For instance, the book “Anintroduction to Digital Video” by John Watkinson, Focal Press, Woburn,Mass., 2001, provides a description of many of the details of knownvideo technology.

The multiplexed signal may be read, for instance, from a computer devicecontrolled memory or storage device 2703 as shown in FIG. 27. A DSP 2802may read 2801 at a high speed, disassemble the correct individualsignals and provide a correct playable signal to outputs 2803, 2804 and2805, which are inputted to displays 2808, 2809 and 2810 respectively.The signals outputted on 2803 are shown as CH1_1, CH1_2 and CH1_3. Thisreflects that a playable signal for a display may require processing ofat least 3 related multiplexed channel signals in the multiplexedsignal. The same applies of course to 2804 and 2805. However, the speedof the DSP combined with available memory is such that the DSP canprocess the multiplexed signal and output three individual videoprograms to displays. In one embodiment, one may provide such animplementation on a PC graphics card with multiple outputs such as 3video program outputs to 3 different monitors.

It should be clear that the ability to concurrently display 2 or morevideo programs at the same time may greatly enhance a multi-mediaexperience. In one example, such a multiplexed program may show a scenewhich is recorded and multiplexed along a beach, wherein one videosignal shows the sea, a second program shows the beach line and a thirdprogram shows the dunes, all recorded at the same time.

One may also have a multiplexed signal that contains 2 or more videoprograms that have scenes that take place at the same time but are atdifferent locations. Those scenes may be related. Thus the displayedmultiplexed programs may show a story line that develops at for instancedifferent locations.

In a further embodiment one may have a multiplexed signal thatrepresents a broad scene, which would be impossible to record from ashort distance by a single camera/lens combination without distortion,but may be recorded by 2 or 3 or more cameras.

In a further embodiment, the video programs may be 2 or more videoprograms that are part of a video game, so they can be displayed at thesame time. For instance, First Person game programs like Quake of IdSoftware of Mesquite, Tex. are very popular. They show scenery from afirst person perspective and/or like a person represented by a camerawould see. A person may turn in a certain direction which is displayedon a monitor like turning of a virtual camera. A person in such a gamegenerally moves in the direction of the camera. Camera direction andspeed of moving are controlled by a game controller. Because of the useof a single monitor and a single virtual camera this provides aone-dimensional view of reality. In reality, a person may look left,right, behind, up and down to scan what is going on and focus on areasof interest. The limited view of a single virtual camera limits theexperience of a gamer. For instance, in a game a person who is theplayer may move forward in a hall way in a building to a group ofpeople, who may be enemies. On the left side of the person a door mayopen. On the right side a door to a room is open showing a second groupof people who may be enemies. In reality, a person may move back quicklyscanning left, right and front and even looking back keeping an overviewand reacting to an immediate event at any of the scanned locations.Presently, with one camera view, there is a limited possibility ofmoving back and viewing three or more locations at the same time.Accordingly, the methods and systems provided herein as aspects of theinvention greatly enhances the sense of reality of a video game byproviding 2 or more concurrent views of one of more scenes fromdifferent perspectives or positions.

A video program in a game is in generally a series of graphics filling ascreen or a frame which is generated by a graphics engine and which maysimulate a 3D environment. The graphics representing a 3D environmentand generated by a graphics engine are displayed on a display. Thegraphics engine calculates aspects of a scene, which may includeperspective, lighting, collisions, objects and other graphics aspects ata rate that provides in display an effect of a real-time 3D environment,not unlike an animated movie or video. The technology of graphicsengines is known and is for instance disclosed in U.S. Pat. No.6,961,055 to Doak et al. and issued on Nov. 1, 2005 which isincorporated herein by reference. The graphics engine is required tocalculate sufficient frames to provide the fluid movie effect. Ingeneral, the graphics engine calculates the images that can be viewedwithin the field of view of a virtual camera representing the firstperson. Such a view may be limiting and may not provide a fullyimmersive 3D experience.

A stereoscopic horizontal field of view of a person may be between 100degrees and 120 degrees. With a moving head to left and right a personmay have a near real-time field of view of greater than 180 degrees. Aperson in a game may be provided a

There may be different embodiments for a graphics engine to calculatethe images. In a first embodiment, a graphics engine may calculate thecomplete wide field of vision scene. This may, for example, involve afield of vision of 210 degrees. A calculated screen or single image of aseries of images that would form the video image may then be dividedinto three displayable images. Each image may then be provided to adisplay. As an intermediary step the graphics engine may store a signalrepresenting each of the images to be displayed in a contiguous mannerin a memory or storage medium. As was shown before, this allows thethree images to be read from memory or storage to be displayedconcurrently and in a synchronized fashion.

In another embodiment, one may instruct the graphics engine to onlycalculate the wide field of view image when multiple displays areavailable. If only one display is available an instruction may beprovided to the graphics engine to calculate only a single camera viewwith a limited field of view.

In yet another embodiment, one may instruct the graphics engine tocalculate three or more screens each related to a view of a virtualcamera in a 3D model of one scene of the environment. An embodiment forrendering two or more independent images to be displayed on two or moredisplays is provided by U.S. Pat. No. 6,885,374 to Doyle et al. issuedon Apr. 26, 2006 and which is incorporated herein by reference. Aprogram or a graphics engine of a video game in general has the model,sometimes called the map, of a game environment having all the elementsthat can be shown on a display. It may contain lighting models, objectmodels of objects in the environment. A map or a situation of a 3Denvironment may for instance be stored in a memory or storage medium. Ifa player revisits part of a map it may thus retrieve the map to show howit was left by the player. A graphics engine may thus use or re-useearlier calculated or already established elements of a screen.Accordingly, by using and/or re-using earlier calculated elements thecalculation of a screen that is related to another screen which may be ascreen related to a center vision virtual camera of one scene and thusis not independent, may be easier to perform by a graphics engine.

As an aspect of the present invention two or more related video programsare acquired and are played are played concurrently on two differentdisplays. It is also provided as an aspect of the present invention thatthe two or more video programs may be recorded concurrently. The videoprograms may be combined in a multiplexed signal and stored on a memoryor storage medium. The programs may be retrieved and played on a displayand may be displayed concurrently and synchronized in accordance with afurther aspect of the present invention. In a game or a video displaythis may provide an effect of immersion and of having a very wide fieldof vision.

A stereoscopic horizontal field of view of a person may be between 100degrees and 120 degrees. With a moving head to left and right a personmay have a near real-time field of view of greater than 180 degrees. Aperson in a game may be provided a wider field of view of the gameenvironment by displaying concurrently and synchronized two videorepresentations that cover a larger field of view than normally providedin a game.

In one embodiment, the displays for concurrent and synchronized displayof the two or more screens related to a scene in a computer game areseparate displays. It is known that one can combine different videoimages and display these combined, such as side by side on one videoscreen. In accordance with a further aspect of the present invention, amultiplexed signal containing two or more video programs are displayedconcurrently and synchronized on a single display screen.

In a preferred embodiment of the present invention, display signals fordifferent displays are stored in a contiguous and multiplexed way in amemory, a buffer or a storage medium, so that playing of the differentdisplays can take place by substantially un-interrupted or jump-freereading of the memory, buffer or storage medium. Jumping in this contextis intended to mean jumping between substantially not contiguous memoryaddresses or storage locations. Jumping may create an interruption orpause which may be noticeable in writing a signal to a display or inswitching between displays.

A game having the capabilities to show multiple scenes concurrently andsynchronized after possibly being read from a storage or memory mediumprovides a player a capability to move a person in a game smoothly andprovide the player with a wide field of vision, and increase a nearreal-time field of vision by allowing a person to look from left toright. To provide a further realistic experience a game may beimplemented having a controller that allows a person to look around andmove at the same time. A program of a video game in general has themodel, sometimes called the map, of a game environment having all theelements that can be shown on a display. A diagram of such a system isprovided in FIG. 35. A user 3500 may play a game displayed at 5 displays3503, 3504, 3505, 3506 and 3507. The game is generated by a programcontaining a graphics engine and generating display signals in acomputing device 3502. The user may use a controller 3501 connected to3502. This controller may be a joystick allowing the user to move withinthe 3D environment generated by the game. The user may move by movingthe controller along the direction of the vertical and horizontal arrowsin 3501. A user may also change a view of a scene by rotating thecontroller around its vertical axis in a clockwise or counter clockwisedirection as provided in 3501.

A graphics engine calculates the actual image that will appear on adisplay. In general, there is no need for a graphics engine to calculatethe elements that are realistically not visible to a camera representingthe player. In a further embodiment of the present invention, a graphicsengine may calculate in a game for display a scene of a wide field ofview beyond that of a single camera. One may display a wider field ofview in the game on a display or on multiple displays. In a furtherembodiment, such a game having a wide field of vision may have acontroller that controls movement of a person in a game and thedirection of a camera in a game.

One controller having such capabilities is, for instance, disclosed inU.S. Patent Application 20070035516 to Voto et al. published on Feb. 15,2007. This allows a person to look left and right in a game by rotatinga joystick and moving in a game by moving the joystick in the directionof movement. In accordance with a further aspect of the presentinvention, a gaming system may also have multiple programs beingdisplayed concurrently and synchronized. This allows a person in a gameto look in one direction, while moving in another direction. Theconcurrently and synchronized displayed video programs may be retrievedfrom a storage and memory medium. The concurrently and synchronizeddisplayed video programs stored on a storage and memory medium may bestored as a multiplexed signal. They programs may also be retrievedindividually from a storage and memory medium and stored in such asfashion inside a playing system that they can be played concurrently andsynchronized.

Embodiments of Storing Multiple Video Programs

In another embodiment of the present invention, one may store individualvideo programs or screens generated by a graphics engine in anon-multiplexed manner on a storage medium or a memory medium or abuffer. The multiplexing function as described for instance in FIG. 25may then be performed for instance as a pre-processing step to themethod as described in FIG. 27. The reading of the signal of 2701 willthen be the reading of individual signals. The system 2702 will combinethese signals as time-multiplexed and contiguous signals and store themin for instance a memory 2703. The demultiplexing and playing may occurin accordance with aspects of the invention disclosed herein.

In a further embodiment, a system is provided that may receive amultiplexed signal such as a QAM modulated signal that contains severalvideo programs that may not be time multiplexed is shown in FIG. 29. Twoor more of these signals may be synchronous and may be video programsrepresenting recording or registration of a scene from differentlocations. A signal 2912 which may be a QAM signal is received, isdemodulated by demodulator 2900 which may also digitize the signals. Forinstance, 3 related digital video signals 2915, 2916, 2917 are providedto a computing device 2902. If the signals are not in a time multiplexedcontiguous format the device 2902 may apply a multiplexer 2918 to createa time multiplexed signal and may store it in memory or storage device2903. Playing of the multiplexed signal may take place by demultiplexingunit 2911 which may have a demultiplexer 2906 and a signal conditioningunit 2907 to provide signals to displays 2908, 2909 and 2910. The unit2907 may also work in accordance with a demultiplexer and DSP asdescribed in relation to FIG. 28.

In a further embodiment, a time multiplexed signal 2912 may be received.The signal may need to be demodulated and then provided to computingdevice 2902. It should be clear that demultiplexing in that case is notrequired. If the signal needs to be stored it should be stored in acontiguous way so that it can be played in substantially a jump-freeway.

In accordance with a further aspect of the present invention, a camerais provided that may record 2 or more video images concurrently. Adigital video camera that captures a video image and stores at leastpart of the image in a memory is known and is for instance described inU.S. Pat. No. 5,343,243 to Maeda and issued on Aug. 30, 1994. Digitalsignal processing of a video image captured by a digital video camera isalso known and is for instance disclosed in U.S. Pat. No. 5,568,192 toHannah and issued on Oct. 22, 1996. Both patents are incorporated hereinby reference.

FIG. 30 shows in a cross-sectional diagram from above a digital videocamera 3000. It has optics such as lenses 3001, an image sensor 3002 anda storage or memory unit 3003 which may also have processingcapabilities. The camera also has an output 3004 which may provide thestored video image to the outside world. Up to recently, the memory of3003 would only serve a buffer function. However, the capacity of memoryis now such that the memory may be in the order of Gigabyte capacity. Ifrequired, the video image may also be provided almost immediately via3004 to a storage medium such as a storage disk. Such a storage mediummay be located inside the body of the camera.

FIG. 31 shows a diagram of a video camera in accordance with an aspectof the present invention. A camera in accordance with an aspect of thepresent invention is able to record concurrently at least two videosignals. The camera, as shown in diagram in FIG. 31, is able to record 3video signals. It has a body 3100 which has three lenses 3101, 3102 and3102 which may capture images of a scene concurrently from a differentperspective. The lenses in this case have an angle of about 90 degrees.Lenses may also have a smaller angle with respect to each other. Acamera may also have more than 3 lenses. Each lens has associated withit a sensor which captures a video image and may store an imagetemporarily in a memory. Lens 3101 has sensor 3106 and memory 3107. Lens3102 has sensor 3109 and memory 3110. Lens 3103 has image 3112 andmemory 3103. Lens 3101 also has a mirror 3104 to reflect an image on thecorrect sensor. Lens 3103 has mirror 3105. Mirrors may not be requiredif sensors are correctly aligned with lenses. Each memory has an outputfor reading the content of the memory. These outputs are 3108, 31011 and3114. A memory may provide a signal when it has for instance a fullframe available for downloading. This may allow an external device toread the memories for instance on a sequential basis: for instance firstmemory 3107, then 3110 and then 3113 to start with 3107 again. One maydo this in such a way that a time multiplexed signal is formed or can beformed from the downloaded image signals. The dotted lines in thediagram indicate a path of a light ray.

FIG. 32 shows a video camera in a further embodiment in accordance withan aspect of the present invention. The camera has a body 3200. Itslenses, mirrors, and sensors may be identical to the embodiment of FIG.31. However, rather than providing individual video signals to theoutside world, the embodiment of FIG. 32 may create its own multiplexedsignal which may be provided to the outside world. Memories 3203, 3204and 3205 which hold a captured image from a sensor may also have someprocessing capabilities to prepare a signal for multiplexing. A circuitmay be under control of a clock signal, thus allowing a circuit to waitfor its turn before it generates a signal. When it is a memory circuitsturn to provide a signal, it provides its signal to a multiplexer 3201.The multiplexer 3201 may also have the capability to provide additionalprocessing, including error correcting coding, image registration andproviding synchronization marks. The multiplexed signal may then beprovided on an output 3202. In a further embodiment the multiplexedsignal may be stored on a storage device inside the body of the camera.Such a storage device may be a memory or a storage medium such as amagnetic disk or an optical disk, or any other storage device that canstore the multiplexed signal. In a further embodiment the storage devicemay be removable and/or unpluggable from the body of the camera.

Accordingly, a camera is provided that can provide concurrent andsynchronized signals which can be time multiplexed or are already timemultiplexed. Further processing of the signals outputted by such acamera can then take place in accordance with one or more aspects of theinvention as disclosed herein.

The camera with a configuration as provided in FIGS. 31 and 32 cannottake a full surround picture because the body of the camera does notallow for looking backward. A further embodiment of a camera to take aseries of full surround concurrent videos is provided in FIG. 33. Thebody 3301 herein as an illustrative example is in a hexagon. This allowsthe positioning of 6 video lenses 3304, 3305, 3306, 3307, 3308 and 3309to capture 6 video images at substantially the same time. Other shapesof the body and different number of lenses are possible and are fullycontemplated. The view of this camera is from above.

A line 3303 is shown through the camera from lens 3304 and lens 3307. Across sectional diagram of the camera viewed from a side perpendicularto line 3303 is shown in FIG. 34. It shows body 3301 and lenses 3304 and3307. The lens 3304 is associated with a sensor 3410. Lens 3307 isassociated with sensor 3408. Sensor 3408 provides a signal that istemporarily stored in a unit 3409, which may have memory and also someprocessing power to condition a signal. Sensor 3410 provides a signal to3411 which has substantially the same function as 3409. Both 3409 and3411 provide a signal representing collected image data to a multiplexer3412, which may provide a multiplexed signal on an output 3414. It is tobe understood that each sensor may also provide its signal which may becollected in a memory first in an individual fashion to a dedicatedoutput.

In each configuration of FIGS. 31-34, each sensor may provide a signalto a common output in a fashion that is controlled by a clock, thuseffectively working as a time division multiplexer. This may work inreal-time under condition of the enabling clock speed complying with thesampling theory. These are merely illustrative and not limitingexamples. Many different configurations are possible to create a timemultiplexed digital signal from a plurality of image and video sensors.

Video cameras may be used in an integrated body. They may also be usedas separate units. In a preferred embodiment two or more video programswhich may be recorded as video images by cameras from one scene fromdifferent views or locations should be played in a concurrent andsynchronized function. These video images may be processed and storedbefore being displayed. A diagram as an illustrative example inaccordance with an aspect of the present invention of an integratedcamera system 3600 is shown in FIG. 36. It is to be understood that sucha system may comprise also 2 or 3 or more than 3 cameras or lenses. Thecamera system has 3 lenses 3601, 3602 and 3603. The system collects thecamera signals from each camera or sensor belonging to a lens. Thesystem may time division multiplex the signals each representing anindividual video program and out put a time division multiplexed signalon output 3604. The intention may be to input this signal to a displaysystem having three displays and showing the three programs concurrentlyin a synchronized way. For instance, the scene may be a sports gamewherein the three lenses provide an overview of the area where a sportis played, such as a stadium.

At the present time of the invention, fairly large flat video displaysare becoming available. As an aspect of the present invention, one mayprovide multiple of these displays to show the multiple video programsas provided by a time division multiplexed signal on output 3604. Thecurrent displays are provided with their own body or encasing. Thismeans that screens as provided in FIG. 35 will not show a seamlessintegrated picture. The rim of each display will clearly prevent suchseamless integration. However, it is still desirable to show on eachdisplay an image that is consistent with the other displays. Suchconsistency means that all displays show images that are what is knownin the art as registered images. This means there is no noticeable jumpin for instance details, size and alignment of images in the details.Registering techniques are known and may be applied as an aspect of thepresent invention. Such registering may be required if one uses aseamless display that includes at least 2 individual displays of whichthe display screens are connected seamlessly.

The easiest way to register images is to record images with cameras thatare synchronized and aligned with each other. The simplest way for amulti-camera system with for instance 3 image sensors is to use threelenses set on focus and other characteristics that will provide imagesthat will appear as registered. However, often one may want to zoom inor out from a detail. This may require that all three lenses zoom in andout in the same way. This may create a change in size of an image withas a result that the three images provided may no longer create aregistered image compared with for instance a pre-zoom position. In oneembodiment, one may create a calibration table for different settingsand position of the lenses, wherein each setting belongs to a differentzoom status and creates a registered 3 images. One may provide thesesettings in a memory and provide the lenses with motorized controls.FIG. 36 shows that lenses 3601 and 3603 may for instance be changed in ahorizontal position. They may also be able to pivot in a plane. One maytake the focus and zoom position of lens 3602 as a lead position andadjust the settings of the other lenses based on the earlier calibrationsettings. One may do that by storing the setting of lens 3602 in amemory; create a registered 3 image display by also adjusting thesettings and relative position of lenses 3601 and 3603; and associatethe position and setting of the lenses 3601 and 3603 with a specificsetting of lens 3602. When lens 3602 is then put in a focus and/or zoomposition, then a processor or a controller may retrieve the associatedsettings of lenses 3601 and 3603 from memory and may drive the lenses3601 and 3603 in the associated position. Thus, one createsautomatically a setting that will generate a registered display ofmultiple image on a display, in this example of 3 images. Preferably,this is done by using motors which will put lenses in their correctrelative position, zoom position and focus position. A change in thesetting of lens 3602 will then result in an automatically correctsetting of lenses 3601 and 3603

In a further embodiment, one may calibrate an integrated camera settingfor a scene for one or more zoom positions. Using such a calibrationmethod allows a camera to zoom in on an object while leaving all imagesregistered if all cameras are for instance motorized and are workingwith a calibration table, which may be stored in a memory. In yet afurther embodiment, one may leave some cameras in a registered imageposition while focusing one camera in a more zoomed position. If one sodesires, by using the calibration table, one may return to a fullregistered situation for all images.

FIG. 37 shows a camera system wherein the cameras are not embodied inone body but may reside on different locations. Cameras 3701, 3702 and3703 then provide their signals to a computing system 3705 which mayprovide a time division multiplexed signal 3704 containing the timedivision multiplexed video programs of the three cameras. As with thesystem as shown in FIG. 36, the cameras and their lenses may bemotorized in order to calibrate the system and create registered imagesfor a range of settings.

FIG. 38 shows a diagram of a true surround video system, wherein aviewer 3800 is completely surrounded by video screens 3803, 3804, 3805,3806, 3807 and 3808. The video programs may be played in real-time by aplayer 3802 in accordance with an aspect of the present invention ofdemultiplexing a multiplexed signal of in this example 6 video programs.Such programs may be recorded with a camera as shown in diagram in FIG.33. Different video surround configurations with more or fewer displaysare possible.

In one embodiment one may use a hard disk drive as a further storagemedium to play signals such as video programs from. A hard disk drivewith for instance a magnetic storage medium may have a storage capacityof over 100 GB and a consistent data transfer rate of up to 125Mbytes/sec. Furthermore, a maximum access time of 5 ms may exist. It maybe that a video program may require a transfer rate of 5 Mbit/sec forplaying in real-time. That means that 3 video programs require 15Mbit/sec of data transfer. Such transfer is well within the limitationsof the transfer rate of a hard disk drive. Three hours of video programsrequires 3*3600*15=162 Gbit z 21 Gbytes of storage. All of this is wellwithin the limitations of the hard disk.

One may use different embodiments for the storing and reading of data.The following is just one embodiment and others are possible. It may bethat the reading speed is too high for direct playing. Accordingly, onemay use one or more buffers to store data retrieved from the hard disk,provide the data to a player and retrieve additional data from the harddisk. With an access time to hard disk data of 5 ms one may for instanceread data for about 900 ms. Assume that video program data can be storedon a hard disk is contiguous blocks of 300 ms of data at a transferspeed of 300 Mbit/sec. That means a block of 0.3 *300=90 Mbit. A blockof 90 Mbit is equivalent to 3 *30 Mbit or at 5 Mbit/sec for real timeplaying 6 seconds worth of data for 3 video programs, which can betransferred from the hard disk within a second. One may thus store one ahard disk in a contiguous fashion a first sequence of data representinga first program for 5 seconds, followed by data for 6 seconds of asecond program, followed by data for 6 seconds of a third program. Thedata of each program are written to a buffer for a player of each videoprogram, a buffer which may be a flash memory of larger than 30 Mbit.Each buffer is read to the player at the required real-time speed of 5Mbit/sec. Around a critical level of for example about 1 second ofunread data in a buffer one may start again a reading process and fillthe buffers.

This is shown in FIG. 39. Three video programs 3901, 3902 and 3903 areprovided to a computing device 3900. The programs may already bedigitized, or they may be in analog form or even modulated and may bedemodulated and may be digitized in a unit 3904. However, the task of3904 is to prepare the three signals for writing in contiguous form to ahard disk or memory 3905. The unit 3904 itself may contain a buffer toachieve the writing. The writing speed to the hard disk is at least anorder of magnitude higher than the transfer speed of the video program.The data representing part of 3 video programs may be stored in acontiguous way on hard drive 3905 for instance in accordance with thediagrams of FIG. 3 a wherein each block of data would be equivalent with30 Mbit of data in the above example. After reading a block it istransferred to the related player which may have a buffer to store acertain amount of data such as 30 Mbit of data. The three players with abuffer are 3606, 3607 and 3608. Each player will decode the storedsignal and provide a playable signal to displays 3910, 3911, and 3912.Each buffer/player may be under control of a clock signal 3909.

One may separate the functions of writing to and reading from a harddisk. For instance one may write the three programs on a removable harddisk in accordance with the above procedure. One may place the removablehard disk or memory in a different computing device for reading andplaying of the three programs.

In a further embodiment one may use a different storage medium forstoring the three programs. For instance one may use an optical disk, amagnetic tape, a mass memory or any other medium that can store blocksof data representing 3 or more video programs that can be read at least3 times the speed of reading for real-time playing of a program. Ingeneral optical disks such as DVDs play at real-time. However, asprovided in the example case of the n-valued or n-state optical disk,the transfer speed and the storage speed of disks may already exceed therequirements for real-time playing of a single video program. It is thenpossible to store at least 3 normal DVD programs on a High Definitionoptical disk and play those disks at least 3 times the single real-timeplaying speed. Recording speeds of 16× are already possible. Thetechnical capabilities of optical disks for storing and playing asdisclosed above and in FIG. 39 are thus already possible. Flash media of30 Mbit are also available. In a further embodiment, one may replace anoptical disk or a magnetic disk with a mass memory, also called asolid-state drive (SSD). Commercial SSDs with a capacity of over 250 GBand very high transfer rates are currently available.

The same reasoning applies to magnetic and to optical andelectro-optical and magneto optical tapes and to any other medium thatmeets the requirements for playing and storing at least 3 video programsin real-time.

While in the illustrative example video programs are used, one may alsoapply aspects of the invention to still images.

It should be clear that the writing speed to media is not as critical ifone does not want to play almost immediately after writing.

In general the embodiments provided herein provide first a demultiplexerand then provide a playable signal to a player for a display. This meansthat each player receives what one may call a base-band or demultiplexedsignal. One may also provide a complete multiplexed signal to a player.This means that each player may also have the capability to demultiplexa multiplexed signal, as is shown in FIG. 11.

The video signals that are provided herein may be accompanied by audiosignals and other signals that may be multiplexed into the final timedivision multiplex signal.

There are many opportunities to process a video program from recordingto replaying. In accordance with an aspect of the present invention avideo program that is to be time division multiplexed, will be sampledand digitized. Digitizing may be in binary form. It may also be innon-binary or n-state form. It also may be compressed. At the point ofmultiplexing the video program is represented by a signal thatrepresents a video quality that meets a certain quality standard. It mayhave a standard DVD quality. It may have a High Definition Videoquality. It may have any other quality standard. A video is assumed torecord an image that can be played in real-time. In accordance with anaspect of the present invention that means that a real-time video showsthe same number of image frames as was recorded. For instance arecording speed for a High Definition TV camera may be 60 Hz or 60frames per second. In accordance with an aspect of the present inventiona multiplexed signal containing a HD quality 60 Hz frame speed may berecorded, stored, retrieved, demultiplexed and reconstructed to play atsubstantially the same quality as it was recorded. Some quality might belost due to noise or errors. But the real-time display of a videoprogram substantially shows the same program as was recorded.

Many video programs are nowadays being watched on small portable deviceof just several square inches to fairly small displays in airplaneentertainment systems for instance. The smaller size of displays meansthat the high number of pixels in signals required for large screens anddisplays are not required for smaller displays. This allows for furthercompression of a video signal and thus a greater storage capacity on amedium. In some displays one may use transfer speeds for real-time videofrom 800 kbit/sec to about 2.5 Mbit/sec. This lower speed than a common5 Mbit/sec may increase the capacity of storing more video programs on amedium.

Mass memories such as flash memories and mass storage such as magneticdisks may have a relatively long random access time, sometimes up to 10ms. However, their serial access time is fast. This means that once thedata is stored in a contiguous areas in high capacity, high transferrate memory or storage medium one may read a stored multiplexed signalat sufficient rates for real-time display of the individually videoprograms embedded in the multiplexed signal. For instance Samsung in2008 announced development of a 2.5-inch, 256 Gigabyte (GB) multi-levelcell (MLC) based solid state drive using a SATA II interface, which itclaimed to be the world's fastest at the time of its announcement. Witha sequential read speed of 200 megabytes per second (MB/s). This issufficient for about 10 programs of 2 hour HD video programs of whicheach requires a transfer rate of about 20 Mbit/sec. Currently, singledrives usually for enterprise purposes are available with spindle speedsof 15K and a transfer rate of over 150 Mbyte/sec for instance fromSeagate.

In accordance with an embodiment of the present invention one may thusmultiplex from external sources, for instance from a transmission of avideo program, a reading of an optical disk, reading from a storage ormemory medium, at least two or at least 3 video programs; one may thenmultiplex those programs into a single time division multiplexed programthat is stored in a substantially contiguous manner on a memory or astorage medium. Substantially contiguous in this context means that in asingle reading cycle sufficient real-time playable signal can berecovered and for instance buffered if required for playing all storedvideo programs, without requiring additional or repeated searching. Thereading device may require a certain access time to find the nextcontiguous sector of data to be read if two sectors are not contiguous.However, it is assumed that the searching and access time for such asector is not longer than the real-time play time of buffered programs.This will have the effect that all programs may be played in a seamlessfashion without noticable interruptions. In a further embodiment a twoor more sectors are stored in a contiguous way.

The above is significantly different from standard DVD formats. Herein avideo program may be recorded from different angles, wherein theprograms may switch between the different angles. However, withoutadjusting the rotation speed of the played DVD it is impossible to playtwo or more angles at the same time in real-time with program qualitythat is the same as playing just one angle in standard quality, whichmay be High Definition (HD) quality. In order to facilitate the playingof two or three or more programs at the same time from a single mediumone thus requires:

1. an ability to record and to store as a concurrently playable digitalsignal two or three or more video programs on a storage medium;

2. an ability to read the two or three or more programs represented in asignal such as a digital signal in a fashion that it meets thelimitations of the sampling theorem;

2. an ability to decode the two or three or more digital sequences intotwo or three or more playable programs; and

3. a display for displaying each of the two or three or more playableprograms.

In a further embodiment one may also display the 2 or 3 or more playableprograms on a single display.

It has been shown how two or three or more video programs can first bestored in a contiguous and multiplexed way on a storage medium that canbe read at a speed that is a multiple of the reading speed required forplaying a single program in real time in standard quality. If thereading speed of the medium is too high, buffers may be used.

An attractive possibility of having multiple programs being displayed atthe same time is to display video programs of one scene or event beingrecorded and displayed concurrently at different displays in such amanner that a wider view of the scene or event is provided. Thisprovides an immersive video experience. The recording of such a scenerequires a special camera or a special arrangement of cameras.

FIG. 40 shows in diagram in accordance with an aspect of the presentinvention a camera that may record at least three video programs thatcan be displayed in such a way that it appears as one continuous videoprogram. It should be clear that one may expand the camera for recordingmore than three video programs. One may also create a camera forrecording 2 video programs. A camera 4000 in this illustrative examplehas at least three lenses 4001, 4002 and 4003. Each lens may have itsown optical sensor. Each lens is also provided with a mechanism to focusa lens. Each lens is also provided with a mechanism to zoom in or tozoom out. Such a mechanism may be a motor like an electrical motor. Thefield of view of a lens is influenced by its zoom. Zooming may bring anobject closer, however, it also makes it field of view smaller. It maybe that one selects an angle between lenses so that fields of view inmaximum zoom still overlap so that one may create one apparentlycontinuous video. However that may also mean that one has a significantfield of overlap at minimal zoom, which means a non-optimal field ofview display. The camera as shown in FIG. 40 can address this problem.One may assume that lens 4002 is the center lens. This means that lenses4001 and 4003 during recording and zooming have to be adjusted to lens4002. This can be done with a coordinated mechanism. A coordinatedmechanism may comprise electrical motors. The lens 4001 has a mechanism4005 and lens 4003 has mechanism 4006. The mechanisms include a zoom inand zoom out mechanism, which may be electrical motors. The electricalmotors may be stepping motors, or any other motor that may but an objectsuch as a lens in a predetermined position. They also include a turningmechanism. Such a turning mechanism cause lenses 4001 and 4003 to turnor rotate toward lens 4002 when the lens 4002 is zoomed in. Theseturning mechanisms may also be driven by motors such as electric motors.The mechanisms 4005 and 4006 also adjust the zoom factor of lenses 4001and 4003. When lens 4002 is zoomed out the mechanisms 4005 and 4006 maycause the lenses 4001 and 4003 to be turned away from lens 4001 toprovide a maximum field of view. The camera provides a signal on output4004.

One way to coordinate the mechanisms is by calibrating the mechanisms onmaximum zoom and minimum zoom and intermediate positions to provide acontinuous picture from the camera on three displays. To combinedifferent images in an aligned way in order to provide a common pictureis called registering images or image registration, which is a knowntechnology. This technology is also known as image stitching. Anoverview of known image registration methods is provided in Zitova,Barbara and Flusser, Jan: “Image registration methods: a survey” inImage and Vision Computing 21 (2003) pages 977-1000, and in RichardSzeliski, “Image Alignment and Stitching: A Tutorial Preliminary draft”,Sep. 27, 2004 Technical Report MSR-TR-2004-92, Microsoft on-line, 2004,which are both incorporated herein by reference in their entirety. Imageregistration is generally applied to still images, to create forinstance an image mosaic or a panorama view. One may also apply imageregistering or registration techniques to create a continuous andcontiguous video image or video image mosaic that is made up from 2 ormore video images. The registration requires that images have someoverlap or at least share an edge. Image registration is able to stitchor connect images to a panoramic image. Image registration techniquesalso can perform task such as image transformation that may correct ormodify lens distortion and parallax effects. Elements of imageregistration techniques are well documented in the literature and theirapplications are fully contemplated and may be applied to all aspects ofthe present invention.

Image registration may include transformation of at least one image toobtain a mosaic that reflects the correct point of view or to correctdistortion by a lens. The matching or aligning of images for imageregistration may be performed by instructions executed by a processorcombined with computer memory.

In one embodiment of the present invention one may create different 3Dvideo images of a scene with a single multi-lens camera or with multiplecameras and combine those into an aligned and concurrently played videoimage providing a panorama view of the scene in a single image.

In a preferred embodiment the video images are 2 or more or 3 or more2-dimensional video images taken by a single camera with multiplelenses. As shown in diagram in for instance FIGS. 36, 40 and 41 a videocamera may have multiple lenses which are held substantially in a knownposition in reference to a body. It is thus fairly easy, by using aknown scene, to calibrate the controlled lens positions and focus and/orzoom and pan with images of the other lens or lenses of the camera. Insuch a calibration case one may take one lens as the lead lens. Thesetting of a lead lens, such as focus and zoom is then associated withcorresponding settings of the focus and zoom of the other lenses, tocreate a stitched or registered multi-image picture. One may store thepositions or settings of the lenses in a memory and associate therequired registration parameters to form one aligned image from eachimage sensor with those lens settings. One may calibrate those settingsfor instance from a close-by scene to a very far (infinity setting)scene. One may include different diaphragm settings, exposure time orshutter speed, zoom settings and/or focus settings to achieve optimalimage registration already in the camera. A processor can thus align theimages already in the camera.

One may also create a set of parameters associated with a camera or lenssetting which may be stored in a memory or may be provided as a separatesignal next to the image signals. One may store the images and relatedparameter settings in a memory or storage in the camera and provide allsignals later to a processing system which may include a display tocreate a registered multi-video image and wherein the system applies thesetting parameters to derive optimal registration.

In one embodiment the camera is a consumer product that may be designedto require almost no user settings. In such a case a set of lenses maybe assumed to go through a pre-determined set of lens settings, whichmay all be pre-set in a memory and/or automatic look-up table and hasfor instance be calibrated during manufacturing. These settings may berelated to for instance a setting of one lens which may be a referencelens as part of multiple lenses. A user may manually provide orinitialize the settings (focus, zoom, exposure time, diaphragm) whichwill be associated and corresponding to settings of the other lenses(which may include also a relative position). Based on the setting ofthe one lens or reference lens the other lenses will be put in thecorresponding settings, for instance by a controller which retrievesthese settings from a memory and based on these settings drives theother lenses in their respective positions to create a registered image.

The one lens may be put manually by a user in a preferred position.However, the one lens may also be automatically put in its preferredposition, for instance by using an autofocus mechanism and a lightdetection mechanism. Accordingly, in a point-and-click embodiment, theone lens goes to its preferred settings based on the conditions anddrags the other lenses with it to the respective related positions,which may result in a registered image. A user may then still manuallyselect a zoom factor for the one lens, which will automatically forcethe other lenses in corresponding zoom settings, based on the initialcalibration settings, so that a substantially registered image may beformed.

One may generate a code based on the settings of the reference lens.Such a code may form an address to a memory. One may also use a table,wherein the inputs are settings of the first or reference lens and theoutput is a code or a memory address. This means that every time thefirst lens is put in a certain position the same code or address isgenerated. The corresponding settings or setting data corresponding tothe setting of a reference lens for the other lenses are then stored atsuch address in a memory and can be retrieved and used by a controllerto put the other lenses in a setting corresponding to the setting of thereference lens.

In a further embodiment the calibration may take place for differentsettings. A first calibration may be for equal zoom for each lens. Asecond calibration may for instance be for a certain lens (such as acenter lens) for having higher zoom than the other lenses. This providessome distortion in registrations that may be pre-set and implemented ina selectable calibration mode. Other selectable calibration modes forregistration in multi-lens video cameras are also fully contemplated.Accordingly, it is not required from a user to perform imageregistration from searching image data; instead parameters that enableregistration operations are associated with a lens/camera setting andmay be used by a processor to automatically generate a registered image,without having to search for a point of registration.

One may in addition implement and use image registering techniquesembedded in a processor or a program that can be executed by a processorin the camera on images that are already substantially registered. Thishas as a result that the camera in general provides a substantiallyregistered video or still image which may be shown on three displays oron one display. In one embodiment wherein only one display is used onlythe image recorded by one lens may be displayed. In a further embodimentone may create a registered set of images and display it as a singleimage on a single display. One may also display the registered image onmultiple displays.

One may also implement registering techniques on a displaying device.Based for instance on a setting of a lens the registering software maylook in a certain area of at least two images to align or register theimages for display on for instance three displays. One may also combinecoordinating mechanisms of lenses with image registering techniques.

In one embodiment one may create a registered image out of the two orthree or more individual images. By applying the settings created duringcalibration one may create images that are either completely orsubstantially registered. Registering software may be applied to tweakor fine-tune the registering process. Because of the earliercalibration, the image may require only minimal adjustment. Forinstance, registering software can be limited to look for matchingobjects or scenes within a limited range of pixels. Such software maylocate optimal registration position by calculating a correlationbetween small areas, varying between for instance about 1 to 10 pixelsin horizontal and/or vertical position of edges of two images that haveto be registered. Such a pixel variation may in a further embodimentalso be about 10 to 25 pixels. Such a pixel variation may in a furtherembodiment also be about 25 to 50 pixels. Such a pixel variation may ina further embodiment also be any pixel variation that allows a processorto determine an optimal registration based on a correlation betweenparts of two images.

One may create an adaptive or learning program that tunes for certainsettings the registration accuracy. One may determine for each newlycalculated registration setting a variation with previous settingsrelated to the calibration setting. If such a change is greater than apre-set limit one may store the newly calculated settings as thestandard or calibrated setting.

Furthermore, it is possible that due to difference in lightingconditions, two corresponding and potentially overlapping areas of twoimages experience different average pixel intensity, such a pixelintensity per color in for instance an RGB color coding. One mayimplement a routine in a controller which calculates for instanceaverage intensities, and will adjust settings to equalize theintensities. Rather than change the settings one may also “filter” oneimage to adjust the distribution of intensities of pixels in one imageto the intensities of the other image to which it is being registered.One may provide a gradual adjustment, so that only the edge or part ofan image is affected. Such adjustments may prevent noticeable edgesbetween images.

In a further embodiment, instead of three lenses in one embodiment onemay provide one or more sets of three or more cameras to record a sceneor an event for display on three or more displays. In such a case onemay have a coordinating mechanism for each set of cameras. For instance,a first set of three cameras that have a certain distance may be used torecord an event over a broad field of vision. Such a broad field ofvision may for instance cover a complete football field to be displayedon 3 or more displays. Another scene may involve a close-up of an eventof for instance a return kick by a player and his environment. This mayrequire a close up by three or more different cameras from the event. Itmay also involve a close-up by a single camera with a single lens. Incase of a recording of a single camera to displayed by three or moredisplays the camera or the device processing the signals to be displayedmay in accordance with another aspect of the present inventionre-calculate the single video image in such a way that three videoimages are created that can be displayed individually on three or moredisplays.

In accordance with a further aspect of the present invention, two ormore video programs are played in real-time using a stored signal orsequence of digital signals from a storage or memory medium, by using asingle reader, the single reader reading a sample for each of the videoprograms within a time period that is smaller than required by thesampling theorem and making a sample for a video program available to aplayer for a video program at a speed that meets the requirement of thesampling theorem. A sample may be any type of sample. It may be a byte,it may be a block representing 3 seconds of video. In a furtherembodiment the samples are stored in a contiguous way on the storage ormemory medium. This means that a storage medium storing for instance 3video programs may read each of the samples of each of the video signalsat less than three times the speed required by the sampling theorem toprovide real-time video display for each of the programs. In yet afurther embodiment, one may require that at least a series of samplesbelonging to the individual video programs are stored in a contiguousway on a storage or memory medium. This may require that the storedsignals are retrieved at a speed significantly higher than dictated bythe sampling theorem. One may buffer the retrieved signals. Time gainedby reading faster may be lost by having to search for the next series ofsamples. One may thus see buffers fill and empty intermittently.

In yet a further embodiment of the present invention, one may storemultiple video signals on an optical disk as a contiguous signal andretrieve the contiguous signal from the optical disk. In yet a furtherembodiment of the present invention one may store and retrieve thecontiguous signal from magnetic disk. In yet a further embodiment of thepresent invention one may store and retrieve the contiguous signal froma memory element. In yet a further embodiment of the present inventionthe memory or storage medium may be removable.

FIG. 41 shows in diagram an illustrative embodiment 4100 of a camerathat can record at least 3 images concurrently of a scene from differentperspectives or angles. The camera may provide a single multiplexedsignal containing the three video signals recorded through 3 differentlenses 4101, 4102 and 4103 and recorded on image sensors 4104, 4105 and4106 and multiplexed through multiplexer 4120. The sensors may beconnected on a network which may be bus controlled by bus controller4110 and may store the image signals on a memory and/or storage medium4112 which is also connected to the network or bus. Further connected tothe network is a camera controller 4111 with its own memory if required.Also connected to the network are three motors 4107, 4108 and 4109 forzooming and moving lenses as required. The motors may be controlled bythe camera controller 4111. Also connected to the network is a processor4113 with its own memory for instruction and/or data storage ifrequired. Furthermore, the network has a control input 4114 forproviding control commands, which may include start recording, stoprecording, focus, diaphragm, exposure and zoom commands. An inputcommand may also include record only with center lens and sensor. Aninput command may also include record with all three lenses and sensors.

The camera also has an output 4115 which provides a signal representingthe instant image of one or of all of the sensors. An output 4116provides the data that was stored in the memory 4112. It should be clearthat some of the outputs may be combined to fulfill the above functions.Furthermore, the camera may have additional features that are alsocommon in single lens cameras, including a viewer and the like. Theseadditional features are fully contemplated.

In a first embodiment, a user may select if images from a single lens orof all three lenses will be recorded. If the user selects recordingimages from all three lenses, then via the camera controller a controlsignal may be provided that focuses all three lenses on a scene.Calibrated software may be used to ensure that the three lenses andtheir control motors are focused correctly. A controller may have accessto a memory that stores related settings. The controller may relate thesettings of a single lens with the desired settings of the other lenseswith the other lens settings being stored in a memory that can beaccessed by the controller. In a further embodiment, the image signalsare transmitted to the memory or data storage unit 4112 for storing thevideo or still images.

In yet a further embodiment the signals from the three lenses may befirst processed by the processor 4113 to be registered correctly into apotentially contiguous image formed by 3 images that can be displayed ina contiguous way. Herein a processor may determine which part of asensor area has to be actively used to create a registered image. Theprocessor in a further embodiment may form a registered image from 3images that may be displayed on a single display. The processor may alsohave a multiplexer that creates a multiplexed signal that is stored.

The processor in yet a further embodiment may also process the images sothat they are registered in a contiguous way if displayed, be it on onedisplay or on three different displays.

In yet a further embodiment, the processor may register the three imagesand multiplex the signals so that they can be displayed concurrently onthree different displays after being demultiplexed.

After being processed the processed signals from the sensors can bestored in storage/memory unit 4112. In yet a further embodiment, thesignals are not stored but are directly provided on an output 4115.

One reason for the different embodiments may be the preference of a userfor a display and to make a camera potentially less complex and/orcostly. One may for instance elect to make sure that all lenses andtheir controls are calibrated as to focus and/or zoom correctly in sucha way that the settings of lenses follow automatically the setting ofone lens. Accordingly, if a lens such as a center lens autofocuses on anobject at a certain distance, and light conditions actuate a certaindiaphragm (or aperture) and/or exposure time (shutter speed) then withthese settings through the earlier calibration step, the settings of theother lenses are associated and may be stored in a memory. As a resultof the actual settings of the center lens, the corresponding settingsincluding focus of the other lenses are retrieved from the memory andused by for instance a controller to

Zooming of all lenses may be coordinated in a similar way. This meansthat zoom of for instance a center lens affects the zoom of the otherlenses, including relative movement of the lenses to create a registeredimage, as field of view may change.

Based on one or more conditions, which may include distance from lens toobject to be recorded, light conditions, zoom conditions, and therelative position of a lens to a reference point, a setting of a lensfor taking an image may include: a focus setting, an exposure time orshutter setting, an aperture or diaphragm setting, and a positionalsetting of the lens in relation to a reference point. In one embodimentone purpose is to create a registered image from at least two or threeimages taken through two or three separate lenses, respectively.However, it is preferable that one creates a registered or panorama typeof images without extensive manual settings of the different lenses.

In one embodiment, it is preferable to have a camera with multiplelenses. One may then assign one lens for instance a center lens if thereare three lenses, as a reference lens. One may use the center lens todetermine the conditions for taking concurrently multiple pictures orrecording multiple concurrent video images that may be registered. Forinstance the center lens may have an autofocus mechanism, a shuttercontrol mechanism, an aperture mechanism and a zoom mechanism. Suchmechanisms may use a stepping motor or a piezo-electric mechanism, suchas marketed by New Scale Technologies of Victor, N.Y. One may apply oneor more sensors to determine distance to object, and lightingconditions. A controller, such as a microcontroller, may determine theoptimal settings and set the operational parameters for the lens focus,aperture and shutter speed. One may use a controller or a memory todetermine an optimal setting for a desired depth of the image. This typeof system for a single lens camera or for a single camera system isknown.

As an embodiment of the present invention one determines a setting for apredetermined condition (an object at a certain distance, a determinedlighting situation, and a desired depth of image) and stores thatsetting in a memory. It is assumed that in this embodiment a camera hasat least 2 lenses and each lens has a related image sensor. The imagesensor data will be stored in image memory or on an image storagedevice, such as memory, magnetic disk, optical disk, or any other devicethat can store image data.

In a further embodiment, the image data is stored on storage medium thatcan store n-state symbols with n>2. An n-state symbol with n>2 means asingle mark or a single signal that can assume one of n>2 states. Sowhen one reads marks from the medium of device or memory a single markwill generate a single signal or symbol. This as opposed to multiplesignals. As an example, an 8-state mark will generate a single signalhaving one of 8 states. This as opposed to an 8-state word of 3 bits.While such a word has one of 8 states, it is stored in general as 3consecutive bits or marks on a memory or storage device. In a furtherembodiment one may write an n-state symbol as multiple p-state marks ona medium. For instance one may write at one write/read position on anoptical disk 2 or more concurrent marks, whereby each concurrent markreflects light at a different wavelength. Reading a mark may thengenerate 2 or more light signals at different wavelength, which can betranslated into for instance a binary word of 2 or more bits.

In a calibration step one also determines the settings of the otherlenses, including a position relative to the reference lens that willcreate on a display or on multiple displays a registered image. One thenstores the settings of the other lenses in a memory in such a way thatthe settings are associated with the setting of the reference lens. Whenthe reference lens detects a condition, it may retrieve from the memorythe parameters to drive the reference by a controller into the settingsrelated to the condition. The controller may then retrieve from thememory the settings of the other lenses corresponding to a setting ofthe reference lens so that when all lenses are put in the settingsrelated to the condition determined for the reference lens all lensesare put in optimal setting to create a registered image.

Thus, one has created a point-and-click camera whereby one referencelens generates the information that allows the creation of a registeredimage. As an illustrative example, one stores settings of the referencelens in a memory and associates the settings of the other lenses withthe stored settings of the reference lens. A controller that drives thesettings of a lens, such as focus, shutter and aperture may do so basedon dynamic input provided by condition sensors. One may program acontroller so that it calculates a setting from sensor data, rather thanretrieve a setting. In a further embodiment, a controller may just applythe settings of the reference lens to the other lenses. In a furtherembodiment, a condition determined for a reference lens may be coded asan address or a reference code that is stored in a memory. One may thenassociate settings of the other lenses with the address or referencecode. One may also associate lens reference codes with the referencecode of the reference lens that allows a controller to generate thesettings that will provide all lenses with appropriate settings for aregistered image. In a further embodiment it may be possible tocalculate the settings of all lenses dynamically based on measuredconditions of a single reference lens.

One may register images in the camera through the processor 4113. If thecorrect settings are selected, the images may already be registered. Onemay then multiplex the individual image signals. However, one may alsoprovide the three images either directly or from memory as parallelsignals to a computing device such as a personal computer. The computingdevice may provide the possibility to select, for instance in a menu,the display of an image of a single lens/sensor. It may also provide aselection to display all three images in a registered fashion. Thecomputing device may then have the means to complete or fine-tuneregistering the images, though by using the calibrated settings no orlittle registering efforts should be required. The computing device maystore the images in a contiguous fashion in a memory or a storage mediumand play the images in a registered fashion either on one display or onthree different displays.

For instance one may provide a signal available on output 4116, whichmay be a wireless output having a radio transmitter that can transmitthe images as a wireless signal to a receiver. Accordingly, a camera maymake 3 or more video images, which may be preferably multiplexed andregistered or may be multiplexed and not completely and only almostregistered available as a radio signal. Such radio signal may bereceived by a receiver and provided to a computing device that canprocess the signals to provide a registered image on a display. Aregistered image may be provided on one display. It may also be providedon multiple displays.

There are different combinations in processing, multiplexing,registering, storing, outputting and displaying. One may elect to domost processing in the camera. One may also do the majority of theprocessing in the computing device and not in the camera. One mayprovide lens setting parameters with image data to facilitate processingby a computing device for processing and consequently displaying theregistered images.

In accordance with a further aspect of the present invention, a systemis provided that is enabled to receive multiple, being at least two ormore and preferably three or more, video signals of a scene and that canbe displayed concurrently on multiple displays. A first embodiment areceiver/display is shown in diagram in FIG. 42 a. A signal, containingthe multiple video signals may be received on input 4201. The multiplevideo signals are multiplexed. They may represent registered images. Theimages may also not be registered. In that case the signal may containcamera and lens parameters that may make registration easier. If theimages are not registered they may be registered by system 4200. System4200 may also be able to demodulate a signal. However, the signal mayalso be already demodulated before being provided on 4201. The system4200 will provide each registered video signal to its own individualdisplay. In the diagram three displays 4202, 4203 and 4204 are shown.The displays will show the registered video images concurrently. In thatcase, the multiplexed signal on 4201 may be demultiplexed in thisexample into 3 individual video signals and processed to be registeredbefore being displayed. If the multiplexed signal on 4201 alreadycontains registered images it may be demultiplexed in 4200 and providedto each corresponding display. One may also provide each display with ademultiplexer that is tuned to a correct channel for the correctregistered video signal.

The advantage of the system of FIG. 42 a is that substantially standarddisplay technology may be used. Each display may have a standard format.Preferably, one should connect displays as seamlessly as possible as todisplay different signals combined as virtually one broad andpotentially panoramic image, thus greatly enhancing a viewingexperience.

FIG. 42 b shows a diagram of a further embodiment. In this case themultiple video images are already demultiplexed and are provided asindividual image signals on individual inputs 4205, 4206 and 4207. Asystem 4208 may demodulate the signals as required and further registerthe images as required and provide individual registered video signalsto displays 4202, 4203 and 4204. Preferably, the images were alreadyregistered and so not substantial registering is required.

Currently, standard video formats are applied to cameras and todisplays. In general a broad panorama type display displaying 2, 3 ormore concurrent video images on one screen may not be available ordesirable. In that case the use of multiple displays may be preferable.However, as part of an enhanced viewing experience one may want todisplay multiple video images seamlessly on one display. This requiresfirst of all one display of the correct format. However, it alsorequires that the 2, 3 or more video images are displayed in one videoframe. Such a video frame should have a width of pixels that is amultiple of a width of a normal frame, while scanning a horizontal lineof a multiple display frame within the same time as a single displaytime. It should be clear that the lines are longer which may beinterpreted that a horizontal line has more pixels.

Such a system is illustrated in diagram in FIG. 43. A signal, which maybe a multiplexed video signal or may represent 3 individual videosignals is inputted on 4301. Accordingly, 4301 may represent 3individual inputs. A computing device 4300 processes the signal orsignals that represent 3 individual video images into one single videoframe which is outputted on 4302, wherein in this case one scene videoimage shown on single display 4303 is made from 3 individual images4304, 4305 and 4306. If required, 4300 may also perform registering theimages, for instance by using camera/lens parameter settings.Preferably, the images were already substantially or completelyregistered by using the aspect of camera lenses calibration as was shownabove. In general a display image is created by consecutive scanning orwriting of horizontal or vertical lines of pixels of the image with onecomplete image being written with a frequency of 60 or 50 Hz.Interlacing may also be applied. Images may be written line after line.Accordingly, it may not be possible in such a system to first writeimage 4304, then 4305 and then 4306 for instance.

In accordance with a further aspect of the present invention a panoramaframe of a video image may be created by multiplexing the correspondinglines of the registered video images. For instance, assume a video imagecontains k horizontal lines of n pixels, wherein each line is written int0 seconds. A combined and registered video image from 3 individualimages may thus have k horizontal lines of 3n pixels, when all lineshave equal length. Each of the lines is formed by writing the n pixelsof a line of the first image in at most t0/3 seconds, the n pixels ofthe corresponding line of the second image also in at most t0/3 secondsand the n pixels of the corresponding line of the third image inconsecutive order, followed by a signal to go to the next line. This isone embodiment to allow displaying an image of k lines of each 3n pixelsin at least t0 seconds per line. This method is illustrated in FIG. 43.An image 4400 is created by first writing pixel line 4401, followed bypixel line 4402 and then followed by pixel line 4403. After 4403 theprocess starts over again with a new line. At the end of completing thefinal line of a frame (which may include interlacing) a signal isprovided for writing the next frame.

It is pointed out that pixels are data storage elements, for instancerepresenting a color intensity. Accordingly, reading pixels may beeasiest and least complex by reading them in consecutive order how theyare stored. Alternatively, one may store pixels of multiple images insuch a way and order that they are consecutively written to a display,for instance in accordance with writing a scan-line by a display.

In a further embodiment one may create 3 different frames from eachindividual image, wherein the part of the line in the combined imagethat is occupied by another image is provided with “no pixel” value.This is illustrated in FIG. 45. Herein, a first frame for a combinedimage 4500 is formed by image 4501 wherein the image is completed by4502 and 4503 all being “no pixels”. Each line should still be ended bya signal that changes the writing to the next line. The same principleis applied to image 4505, which is preceded by empty lines of a blankimage 4504 and succeeded by a blank image 4506. The third image iscreated by preceding image 4509 by blank images 4508 and 4507. Eachimage should be written at most one third of the display frequency.Furthermore, measures may be required to prevent excessive flicker inthe combined image.

In yet a further embodiment of the present invention, one can write asingle panoramic video image by writing each corresponding frame theconsecutive video image as vertical lines, rather than horizontal lines.For instance, a video image that is comprised of three registered videoimages and as shown in diagram in FIG. 44 can be written by firstwriting the consecutive vertical image lines of pixels of image of linefrom pixel 4401 down to pixel 4404, starting with the next line at pixel4405 down to pixel 4406 etc, until all vertical lines of the combinedvideo image frame are written and the next frame can be processed.

Other variations, for instance, using interlacing of frames are fullycontemplated. One may assume panorama images that are full highdefinition quality or at least full standard image or video imagequality. In accordance with a further embodiment, one may show apanorama of 2 or 3 or more images on a single screen or display. In thatcase, the resolution of the image decreases and one may downsample theresolution of an image.

In a preferred embodiment, one should store pixels of multiple images insuch a way that reading of a memory is coordinated with writingscanlines on a display.

While aspects of the invention have been illustrated with video images,it should be clear that the above also applies to photographs or stillimages.

In accordance with aspects of the present invention, systems and methodshave been provided that record three or more video images with a singlecamera 4601 with multiple lenses as shown in FIG. 46. The camera maystore the data of the signal or signals representing the multiple videoimages in a memory or on a storage medium. The camera can at a desiredtime provide a signal representing the video images recorded or sensedby the camera via a connection 4602 to a computing device 4603. Part ofthe processing of the individual images may take place in the camera orin 4603. As a result of the processing a signal 4604 which may bemultiple signals is/are provided to a display 4605 which displays thecombined and registered images concurrently and registered. The display4605 may be multiple displays. The display 4605 may also be a singledisplay. As a result a user of a system as shown in FIG. 46 can enjoy anenhanced video experience by viewing a real-time display of a scene inpanorama view taken by a single camera.

FIG. 47 shows in diagram a system for displaying a panorama image from ascene formed from 3 or more registered video images taken by 3 or moredifferent cameras of a coordinated camera system 4701. By using forinstance 3 or more cameras in different locations a differentperspective of a scene may be provided. The cameras are coordinated andcalibrated in focus, zoom, pan and position by a computing device 4702.A plurality of cameras may be provided of which one is a lead camera.

For instance, a system may have three cameras 4708, 4709 and 4710. Eachcamera has devices and circuitry to record as well as activate settingssuch as focus, zoom, diaphragm, exposure and pan. These settings may becalibrated and used in such a way that the images generated by thecameras will generate a substantially registered panorama image. Tofurther illustrate the coordinated panorama view, each camera isprovided with a position sensor/actuator. Camera 4708 has positionsensor/actuator 4711; camera 4709 has position sensor/actuator 4712; andcamera 4710 has position sensor/actuator 4713. One may use positionsensor/actuators that only work in one plane, for instance in ahorizontal plane. One may also use position sensor/actuators that workin a vertical and in horizontal directions.

One may thus again create a plurality of calibration settings, whereinat different focus setting of a lead camera, for instance camera 4709,the other cameras follow with settings in such a way that a registeredor almost registered combined image will be generated and can bedisplayed. In one embodiment the cameras can pivot and point from afixed position, whereby the settings of 4708 and 4710 are determined bythe settings of camera 4709. In a further embodiment one may place thecameras movably on a rail, still using the settings of camera 4709 asthe lead settings to be used to determine the settings of 4708 and 4710to create a substantially or completely registered combined image.

Such a device may if desired store the images in a memory or a storagemedium. The signals representing the images are then provided on aconnection 4703 to another processing device 4704 which may process theimage into a single signal or multiple signals on a connection 4705 to adisplay unit 4706 to display the registered and concurrent images into asingle panorama video image. The display unit 4706 may comprisedifferent individual displays or it may be a single video display.

The systems of FIGS. 46 and 47 may store either the individual videoimages or the combined and registered video image on a memory or astorage medium before it is displayed. The signal provided on either4604 or 4705 may be provided in real-time or may be provided from memoryor a storage medium.

In a further embodiment, a mobile computing device, which may be amobile phone or a Personal Digital Assistant (PDA) or a Blackberry® typeof device, is provided with 2 or more and preferably 3 or more lenseswith related photo/video sensors which are calibrated to take a combinedand registered image which may be a video image or a still image. Adiagram is shown in FIG. 48 of a mobile computing device 4800 which maycommunicate in a wireless fashion with a network, for instance via anantenna 4804. While the antenna is shown it may also be hidden withinthe body. As an illustrative example the device has 3 lenses 4801, 4802and 4803 which are enabled to record a scene in a way wherein the threeindividual images of the scene can be combined and registered into awide view panoramic image, which may be a video image. The device has acapability to store the images in a memory. The device has a processorthat can create a combined image. The combined image, which may be astatic image such as a photograph or it may be a video image and can bestored in memory in the device. It may also be transmitted via theantenna 4804 or via a transmission port for output 4805 to an externaldevice. The output 4805 may be a wired port for instance a USB output.It may also be a wireless output, for instance a Bluetooth output.

Viewing of the image may take place real-time on a screen 4903 of adevice 4900 as shown in FIG. 49, which may be a different view of thedevice of FIG. 48. For instance FIG. 48 may be a view of the device fromthe front and FIG. 49 from the back of the device showing keypad 4901.In FIG. 49 it is shown that the device is comprised of at least twoparts 4900 and 4905, connected via a hinge system with connectors 4902that allows the two bodies to be unfolded and body 4905 turned fromfacing inside to facing outside. Body 4900 may contains input controlssuch a keys. Body 4905 may contain a viewing display 4903. The lenses ofFIG. 48 are on the outside of 4900 in FIG. 49 and not visible in thediagram. Body 4905 with screen 4903 may serve as a viewer when recordinga panoramic image with the lenses. It may also be used for viewingrecorded images that are being played on the device. The device of FIG.49 may also receive via a wireless connection an image that wastransmitted by an external device. Furthermore, the device of FIG. 49may also have the port 4805 that may serve as an input port forreceiving image data for display on display screen 4903.

A display 4903 in a portable computing device in generally is relativelysmall. While the camera may take full resolution pictures or videos thatare registered, a display 4903 may generally not able to display a highpixel image. Furthermore, one may assume display 4903 to be a singledisplay. Accordingly, the device of FIG. 49 may generate a panorama orregistered image that is derived from (in this case) 3 images. Whilecalibrated, 3 registered high quality images are generated. One may useseveral solutions to display the registered images. One may downsampleeach image so that the downsampled images can be appropriately displayedon 4903. One may also provide a circuit to create from 3 downsampledimages a single image that can be displayed on display. For instance bycombining pixels in (downsampled) lines as illustrated in FIGS. 44 and45. One may store those images in downsampled and correctly image lineformatted format in a special memory or storage medium. One may alsostore the high-quality registered images in a memory or storage mediumin the device. One may store the images in such a way that is known howto address and retrieve the individual lines of the individual images.It is then possible to instruct the device to read the memory or storagedevice in such a way that only the pixels required for display on 4903are read and appropriately displayed on 4903. The reading of the memorythen performs the downsampling and merging of image lines.

In one embodiment one may assume that the surface of the device as shownin FIG. 48 is substantially flat. The camera lenses 4801, 4802 and 4803may be positioned in such a way that they have a combined maximum fieldof view of 180 degrees. This may be sufficient for cameras with 3 lenseswherein each lens has a maximum field-of-vision of 60 degrees. In afurther embodiment, one may have more than 3 lenses, enabling a combinedfield-of-vision of more than 180 degrees. In such a further embodimentthe surface containing the lenses may be curved, allowing 3 or morelenses to be positioned such as to provide a combined field-of-view ofgreater than 180 degrees. In a further embodiment 3 or more lenses mayalso be positioned in such a way that they cover a field of view ofgreater than 180 degrees.

A camera on a mobile phone is often considered a must have accessory.Accordingly, one may prefer a multi-lens camera that will createpanorama type images (either photographs and/or video), preferably atthe lowest possible cost. In such a low cost embodiment one may forinstance apply only a two lens camera. This is shown in diagram in FIG.50 with camera phone 5000 with lenses 5001 and 5002. In diagram in FIG.51 it is shown how scenes are seen by the lenses. Lens 5001 ‘sees’ scene5102 and lens 5002 ‘sees’ scene 5101. A processor in the camera maystitch the images together. The ‘stitching’ may be as simple asprocessing just parts of the images. In general the edge of an image maysuffer from lens distortion. For instance, one may ignore an ‘inner’edge of 10% of the image created by a lens. That part of the image doesnot require a sensor. These parts are indicated by lines 5103 and 5104.In this embodiment one also requires an overlap of the two images. Forillustrative purposes, one may make this overlap also 10%. In a furtherembodiment that overlap may be more; it may also be less than 10%.

FIG. 52 shows a registered and combined image. The image may be aphotograph. It may also be a video image. If it is a video image, otheraspects of the invention such as multiplexing, may be applied to enableconcurrent display of multiple images as a single video image.

In general, one lets software decide how to stitch two or more images.As an aspect of the present invention the settings of the lenses arecalibrated for different settings such as distance, lighting conditionsand in certain cases also for zoom. These settings are related to atleast one measurement that can be performed by the camera, such asdistance for instance with an autofocus mechanism. All settings relatedto such an autofocus for 2 or more lenses are then stored in a memory.Furthermore, one may determine which part in coordinates of pixels andespecially edges of a sensor can be processed to automatically generatea registered image. A processor or a controller, based on a measurementby the camera may retrieve the related settings from a memory. Thecontroller then implements the settings, such as focus, diaphragm, etcto each relevant lens and provides a processor with information whichpart of a sensor to process and to store in a memory in such a way thatthe memory (or storage medium) has stored a completely or substantiallycompletely registered image. No further stitching is then required, orjust minimal processing for position adjustment.

In a further embodiment, one may provide additional instructions, forinstance to make sure that pixels in a bordering region reflect asimilar intensity of grey levels and/or colors so that a smoothtransition occurs during display. This is known as blending. One mayalso apply transformational software that in case of distortion maycorrect the edges of different images to remove or diminish edgeeffects. However, the settings of this software depend completely orlargely on the stored settings or are part of the stored settings, sothat no further adjustment or just very limited adjustment is required.

The embodiment as provided in FIG. 50 and shown in FIG. 52 is unusual inat least one sense that it creates a center of an image by using theedges of the images created by two lenses. In general, as in someaspects of the present invention, one assigns one lens to the center andpresumably an important part of the image. The above embodiment allowsfor creating a good quality image by using inexpensive components andadjusting the quality of a combined image by a set of instructions in aprocessor.

In a further embodiment of the present invention, images in a staticformat, in a video format, in a combined and registered format and/or inan individual format may be stored on a storage medium or a memory thatis able to store a symbol as a non-binary symbol able to assume one of 3or more states, or one of 4 or more states.

In a further embodiment, a combined and registered photograph or videoimage being displayed concurrently has an image quality that issubstantially equal of the image quality of the individual photographsor video images before combining them. For instance, n images with n≧2or n≧3 that are to be combined may each have a quality of not less thanq pixels per mm² or equivalent. The combined photograph or video imageto be displayed in accordance with an aspect of the present inventionthen has an image quality not less than q pixels per mm² or equivalent.In accordance with a further embodiment of the present invention, theimages may be displayed on a single or on multiple displays. In afurther embodiment the combined n images may be downsampled anddisplayed on a single display with the combined image quality of qpixels/mm² or less.

There are several ways or methods to provide a calibration of lenses andrelated image sensors to create automatically a registered image ofmultiple images. For illustrative purposes, a number of these methodswill be described. FIG. 53 shows in diagram three image sensors 5301,5302 and 5303, each sensor belongs to a lens. The sensors are shown ashaving overlap. It is pointed out that this is only shown to illustrate

In a first embodiment the lenses may be put in a fixed position inrelation to each other. That means that the sensors 5301, 5302 and 5303must provide images with overlap for all focus and zoom settings forwhich the camera will be used. That means that one may have to selectdifferent overlap positions for different focus and/or zoom settings. InFIG. 53 the image as shown has a certain amount of overlap. It ispossible that the lenses may have a certain amount of distortion,especially at the edges of the sensor. One may therefore select as theedge of connection between images for instance about the middle of theoverlap. Assume that the diagram of FIG. 53 shows the three images asprovided by the sensors. One may select as the optimal overlap positionand as the connection line, the lines 5307 and 5308. It is pointed outthat these lines may be selected during calibration, but are not realedges. Data manipulation has to create these edges. By setting thoselines one may then store next as a first image, image 5301 generated onthe sensor cut off at line 5307, as image two the part of image sensor5302 cut-off at lines 5307 and 5308; and as image three part of imagesensor 5303 cut-off at line 5308. If there is no distortion at the edgesone may then store each of the reduced images in memory and create aregistered image from the three images. One may create one image fromthree images and display it on one display. One may also display it onthree displays. In case of video images one may multiplex the data tocreate a playable registered video image. One may call a cut-off linealso a merge line. A merge line can be determined by coordinates in animage sensor or can be described as a partition in a memory storingimage data generated by the sensor. In a set of sensors to generate aregistered image sensor data, either directly from the sensor or in amemory, that is on one side of a merge line is used, while data on theother side of the merge line is ignored. A merge line may be a straightline, it may also be a curved line or any other line that separates datato be used in a registered image from data to be ignored for aregistered image.

In a further embodiment, one may experience significant distortion inthe overlap area between images, so that they do not registeradequately. In that case one may determine and implement a pixeltransformation that will create a registered image. The transformationwill remain the same if the lens settings remain the same and may remainin effect for all images that are processed during the setting. One maystore the transformation parameters for registering as part of thesetting in the memory. One may also store the position of the cut-offlines as part of the setting. One may store a setting also as a settingcode that can be retrieved from memory.

One may also store coordinates of the cut-off lines and store the imagedata in full in memory. One may use the coordinates of the cut-off linesto control which part of the stored images will be used to create and/ordisplay a registered image.

It was already explained that display of the registered image on a smalldisplay, for instance, on the camera, requires fewer pixels than in ahigh definition (HD) display and that one may have to downsample thedata. This may mean that fine tuning of registering that is required inhigh definition mode is not required in low definition mode. In oneembodiment it may be that within a certain HD range a misalignment of 5pixels may occur. Such a misalignment may not be noticeable on adownsampled display or on separate HD displays, wherein a display frameprevents exact alignment of the display screens anyway. In those casesfurther tuning of registering may not be required. In a furtherembodiment detailed alignment may not be required in case ofmisalignment of up to 10 pixels. In a further embodiment, a misalignmentof over 10 pixels may not require further registering in case of adownsampled display or multiple HD displays with interfering supportingframes for instance.

In a further embodiment, one may apply adjustment to either the settingof the shutter or aperture or one may adjust the intensity of the pixelsin images based on images in an overlap area. One may for instancecompare the pixels on or in the area of line 5307 in image 5301 and5302. For perfect registration one would like no difference in intensityin the overlap area so that connection is seamless. It may be thatbecause of recording angles of lenses and shadow effects that there is ameasurable difference between pixels on or around the cut-off line. Onemay adjust lens setting to make the transition are as smooth aspossible. Another adjustment mechanism is to adjust in a transition areathe pixels in one or both of the images in such a way that thetransition is seamless. Such filter and transition smoothing operationsare well known. The required parameter settings for this can be includedand stored in the lens settings to be executed during recording or laterduring processing for display.

In a further embodiment, one may provide an image selection tool thatallows the setting of an active registered image box that tells aprocessor which part of a register image should be displayed. One mayuse this setting to save only data within the box. One may also use thesetting to determine which part of the stored data will be used andprocessed to be displayed. For instance, in FIG. 54 all elements exceptthe active box are identical to the ones in FIG. 53. However, comparedto active box 5309 in FIG. 53 in FIG. 54 the active box has been reducedto 5409, so that in this instance objects 5304 and 5306 will not bedisplayed in the registered image.

FIG. 55 shows in diagram yet a further embodiment, using three imagesensors 5501, 5502 and 5503. Herein, either the lenses related to 5501and 5503 and/or the lens/sensor unit related to 5501 and 5503 can bemoved or rotated, so that the images have more or less overlap based onthe position of the units. For instance, in a certain position asprovided in FIG. 55 the width of the registered image may be sufficient,while the selected position of cut lines 5507 and 5508 assure perfect oralmost perfect alignment without further need for registrationtransformation and/or pixel adjustment. In that case the relativeposition of units corresponding to 5501 and 5503 should also be storedas a calibration setting and be associated with settings related to thesetting of 5502.

It should be clear that if one uses one display for displaying theregistered image it is not required to multiplex the multiple images. Inthat case one may determine the required format of the registered imageand create an image with the appropriate pixels and image lines that canbe written on a single display. This applies for still images as wellfor video images. One may create such a single registered image in thecamera. One may also create it at the display device. This allows a userto select a display mode. For instance one may select a mode wherein oneof the images is displayed on a single display. In a second mode asingle registered image is displayed on a single display. In a thirdmode, two or more multiplexed and registered images are displayed on twoor more displays.

In a further embodiment, one may send a registered image made from twoor more images to a display device. This registered image may be in HDformat. The display device may be set in a mode to select displaying ona single or on multiple displays. The display device may be providedwith means to break up the registered image in individual images or todownsample such an image.

In accordance with a further aspect of the present invention, a combinedand concurrently displayed image of a scene created from n images isprovided to an apparatus as a multiplexed signal created from nindividual images.

In accordance with a further aspect of the present invention, a combinedand concurrently displayed image of a scene created from n images istransmitted to an apparatus as a multiplexed signal created from nindividual images.

In accordance with a further aspect of the present invention, a combinedand concurrently displayed image of a scene created from n images isstored in a storage device as a multiplexed signal created from nindividual images.

In accordance with a further aspect of the present invention, a combinedand concurrently displayed image of a scene created from n images isdisplayed as n demultiplexed signals created from a single multiplexedsignal.

In accordance with a further aspect of the present invention, a combinedand concurrently displayed image of a scene created from n images has anaudio signal associated with each image.

Static images and video images that are combined and registered inaccordance with one or more aspects of the present invention arerecorded at the same time or substantially at the same time.Substantially, at the same time herein means that within therestrictions of the sampling theorem the images are recorded within anassigned time slot in accordance with the sampling theorem. It was shownabove that the signals representing individual images taken by differentlenses have to be multiplexed. Theoretically, one may say that a signalat the beginning of a time slot and a signal at the end of a time slotrepresent images taken at different times. However, in the presentcontext, differences in time between images that fall within a time slotfor display of these images, such images may be considered to be takenat the same time.

The methods provided herein as an aspect of the present invention can beimplemented in a processor, such as a microprocessor. It can also beimplemented in a processor such as a digital signal processor.Instructions to be executed to provide the steps of the methods asdisclosed herein can be stored in a memory. Part or all of the aspectscan also be implemented in dedicated or customized circuitry. It canalso be implemented in programmable logic such as a Field ProgrammableGate Array (FPGA). Memory devices and storage devices as provided hereinmay be binary devices. They may also be nonbinary device. The methods asprovided herein may also be implemented in nonbinary switching devices.

A controller may be a microcontroller such as a programmablemicrocontroller. These controllers that take input from external sourcessuch as a sensor and drive a mechanism based on such input and/orprevious states are known. Controllers that control aspects of a camera,such as focus, zoom, aperture and the like are also known. Such acontroller is for instance disclosed in U.S. Pat. No. 7,259,792 issuedon Aug. 21, 2007 and U.S. Pat. No. 6,727,941 issued on Apr. 27, 2004which are both incorporated herein by reference in their entirety. Sucha controller may also be known or be associated with a driving device.Such a driving device is for instance disclosed in U.S. Pat. No.7,085,484 issued on Aug. 1, 2006, U.S. Pat. No. 5,680,649 issued on Oct.21, 1997 and U.S. Pat. No. 7,365,789 issued on Apr. 29, 2008 which areall 3 incorporated herein by reference in their entirety.

While there have been shown, described and pointed out, fundamentalnovel features of the invention as applied to preferred embodimentsthereof, it will be understood that various omissions and substitutionsand changes in the form and details of the methods, systems and devicesillustrated and in its operation may be made by those skilled in the artwithout departing from the spirit of the invention. It is the intention,therefore, to be limited only as indicated by the scope of the claims ortheir equivalents appended hereto.

The following patent applications, including the specifications, claimsand drawings, are hereby incorporated by reference herein, as if theywere fully set forth herein: (1) U.S. Non-Provisional patent applicationSer. No. 11/042,645, filed Jan. 25, 2005, entitled MULTI-VALUEDSCRAMBLING AND DESCRAMBLING OF DIGITAL DATA ON OPTICAL DISKS AND OTHERSTORAGE MEDIA; (2) U.S. Pat. No. 7,218,144 issued on May 15, 2007,entitled SINGLE AND COMPOSITE BINARY AND MULTI-VALUED LOGIC FUNCTIONSFROM GATES AND INVERTERS; (3) U.S. Pat. No. 7,397,690 issued on Jul. 8,2008, entitled Multi-Valued Digital Information Retaining Elements andMemory Devices; (4) U.S. Non-Provisional patent application Ser. No.12/061,286 filed on Apr. 2, 2008, entitled Multi-State Latches Fromn-State Reversible Inverters; and (5) U.S. Non-Provisional patentapplication Ser. No. 11/964,507 filed on Dec. 26, 2007, entitledIMPLEMENTING LOGIC FUNCTIONS WITH NON-MAGNITUDE BASED PHYSICALPHENOMENA.

The invention claimed is:
 1. A mobile computing device, comprising: asingle body camera, containing a first lens with a first image sensorwith a first active sensor area and a second lens with a second imagesensor with a second active sensor area; a memory containing a firstcamera setting, including a determination of a portion of the firstactive sensor area and a portion of the second active sensor area to beactively used to create a registered image, wherein the portions are notthe whole first active sensor area and the whole second active sensorarea and the determination includes the portions in coordinates ofpixels and edges of the first and second active sensor areas toautomatically generate the registered image, wherein both the portion ofthe first active sensor area and the portion of the second active sensorarea determined during a first calibration step and all image data of asingle image frame generated through the first lens by the portion ofthe first active sensor area of the first image sensor from a scenemerged with all image data of a single image frame generated through thesecond lens by the portion of the second active sensor area of thesecond image sensor from the scene form a registered calibration imageof the scene; a controller, configured to retrieve during an operationalstep of the single body camera the first camera setting from the memory;a multiplexer configured to multiplex all the image data of the singleimage frame generated through the first lens by the portion of the firstactive sensor area of the first image sensor and all the image data ofthe single image frame generated through the second lens by the portionof the second active sensor area of the second image sensor; aprocessor, configured to store on an image memory based on the firstcamera setting during the operational step the multiplexed image datagenerated by the portion of the first active sensor area of the firstimage sensor and image data generated by the portion of the secondactive sensor area of the second image sensor as contiguous image datarepresenting the registered image; and a display enabled to display theregistered image generated from the contiguous image data.
 2. The mobilecomputing device of claim 1, the single body camera further comprisingat least a third lens with a third image sensor with a third activesensor area and wherein the first camera setting in the memory includesa determination of a portion of the third active sensor area of thethird image sensor that is smaller than the whole of the third activesensor area of the third image sensor.
 3. The mobile computing device ofclaim 1, wherein the first lens is a reference lens and a setting of thesecond lens during the operational step including the portion of thesecond active area of the second image sensor is determined by a settingof the first lens during the operational step.
 4. The mobile computingdevice of claim 1, wherein the registered image is a video image.
 5. Themobile computing device of claim 1, further comprising a mobile phone.6. The mobile computing device of claim 1, wherein the registered imageis a panoramic image.
 7. The mobile computing device of claim 1, whereinthe registered image is a stereoscopic image.
 8. The mobile computingdevice of claim 1, wherein the registered image is created fromdownsampled image data generated by the portion of the first activesensor area of the first image sensor and downsampled image datagenerated by the portion of the second active sensor area of the secondimage sensor.
 9. The mobile computing device of claim 1, wherein thefirst camera setting is determined by at least a first setting of anautofocus mechanism and the controller places the first and second lensin a focus setting determined by at least the first setting of theautofocus mechanism.
 10. The mobile computing device of claim 1, whereinthe first camera setting is associated with a first zoom setting of thefirst lens and the controller places the second lens in a zoom settingdetermined at least by the first zoom setting of the first lens.
 11. Themobile computing device of claim 1, wherein the registered calibrationimage is registered within a range of 25 pixels or less.
 12. The mobilecomputing device of claim 1, wherein a processor transforms image datagenerated by the first and the second image sensor to correct lensdistortion.
 13. The mobile computing device of claim 1, wherein thedisplay includes a first display displaying an image created from imagedata generated by the portion of the first active sensor area of thefirst image sensor and a second display displaying an image created fromimage data generated by the portion of the second active sensor area ofthe second image sensor.
 14. The mobile computing device of claim 1,wherein the first camera setting includes a first focal setting of thefirst lens and a second camera setting includes a second focal settingof the first lens and another portion of the first active sensor area ofthe first image sensor.
 15. The mobile computing device of claim 1,wherein the processor is configured to read a first line of image datagenerated by the portion of the first active sensor area of the firstimage sensor and a first line of image data generated by the portion ofthe second active sensor area of the second image sensor to generate bythe multiplexer image data for a first line of the registered image onthe display.
 16. A device to display a registered image of a scene,comprising: an image data storage device enabled to contain image dataof the scene concurrently recorded during a camera operation through afirst lens and generated by a portion of a first active area smallerthan a whole of the first active area of a first image sensor andthrough a second lens and generated by a second active area smaller thana whole of the second active area of a second image sensor, wherein theportions of the first and second active areas are determined during acalibration, wherein all image data of a single image frame generatedthrough the first lens by the portion of the first active sensor area ofthe first image sensor from a scene merged with all image data of asingle image frame generated through the second lens by the portion ofthe second active sensor area of the second image sensor from the sceneform a registered calibration image of the scene, the determinationincluding in coordinates of pixels and edges of the portions of thefirst and second active sensor areas to automatically generate theregistered calibration image and all the image data of the single imageframe generated through the first lens by the portion of the firstactive sensor area of the first image sensor and all the image data ofthe single image frame generated through the second lens by the portionof the second active sensor area of the second image sensor aremultiplexed by a multiplexer before being stored on the image datastorage device as contiguous image data representing the registeredimage; a processor enabled to read the contiguous image data from theimage storage device; and one or more display screens enabled to receiveimage data from the processor to display the registered image of thescene.
 17. The device of claim 16, wherein the device contains at leasttwo display screens to display the registered image of the scene. 18.The device of claim 16, wherein the device is a mobile computing devicewith a camera and a wireless connection to a network.